2024-09-15T02:33:23.437 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T02:33:23.454 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:33:23.536 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} email: skanta@redhat.com first_in_suite: false job_id: '7905170' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBFdla0bUvc8hN2PIJ0soM2p00rcrdYe3ECnNfHVq8kVgROECIA+yJMp4NlwQ8u6EYK1AbbjMx7RlNHLi4/smk= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBzgR2t8fqyln5yFzb62a4CsxCsWAtIlw4T6Sfxjzjif4x3ZZbLl+oUtA005ZN63cWRKKiQxm/GFsVpJZXwh4+k= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T02:33:23.536 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T02:33:23.537 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T02:33:23.537 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T02:33:23.538 INFO:teuthology.task.internal:Checking packages... 2024-09-15T02:33:23.560 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T02:33:23.560 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T02:33:23.560 INFO:teuthology.packaging:ref: None 2024-09-15T02:33:23.560 INFO:teuthology.packaging:tag: None 2024-09-15T02:33:23.560 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:33:23.560 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:33:23.560 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:33:23.709 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T02:33:23.711 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T02:33:23.717 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T02:33:23.717 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T02:33:23.752 INFO:teuthology.task.internal:Saving configuration 2024-09-15T02:33:23.764 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T02:33:23.796 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T02:33:23.815 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:28:25.785199', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBFdla0bUvc8hN2PIJ0soM2p00rcrdYe3ECnNfHVq8kVgROECIA+yJMp4NlwQ8u6EYK1AbbjMx7RlNHLi4/smk='} 2024-09-15T02:33:23.832 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:28:25.783410', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBzgR2t8fqyln5yFzb62a4CsxCsWAtIlw4T6Sfxjzjif4x3ZZbLl+oUtA005ZN63cWRKKiQxm/GFsVpJZXwh4+k='} 2024-09-15T02:33:23.832 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T02:33:23.840 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-15T02:33:23.841 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-15T02:33:23.841 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T02:33:23.933 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fae5d67bac0>, signals=[15]) 2024-09-15T02:33:23.933 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T02:33:23.939 INFO:teuthology.task.internal:Opening connections... 2024-09-15T02:33:23.939 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:33:23.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:33:24.014 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:33:24.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:33:24.093 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T02:33:24.100 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-09-15T02:33:24.117 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-09-15T02:33:24.117 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-09-15T02:33:24.174 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-09-15T02:33:24.175 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T02:33:24.176 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T02:33:24.176 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T02:33:24.176 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-09-15T02:33:24.195 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-09-15T02:33:24.213 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-09-15T02:33:24.214 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-09-15T02:33:24.272 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-09-15T02:33:24.272 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-09-15T02:33:24.272 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-09-15T02:33:24.272 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T02:33:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T02:33:24.274 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-09-15T02:33:24.295 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T02:33:24.304 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T02:33:24.335 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T02:33:24.336 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:33:24.339 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:33:24.355 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T02:33:24.385 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T02:33:24.385 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:33:24.396 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:33:24.482 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T02:33:24.528 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T02:33:24.542 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T02:33:24.542 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T02:33:24.543 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:33:24.543 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:33:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:33:24.560 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-09-15T02:33:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:33:24.561 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-15T02:33:24.618 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-503.el9.x86_64 2024-09-15T02:33:24.618 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-503.el9.x86_64 2024-09-15T02:33:24.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-09-15T02:33:24.619 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-503.el9.x86_64 2024-09-15T02:33:24.620 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-503.el9.x86_64 2024-09-15T02:33:24.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-09-15T02:33:27.088 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-15T02:33:27.408 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.2 MB 00:00 2024-09-15T02:33:29.800 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 16 MB/s | 20 MB 00:01 2024-09-15T02:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 20 MB 00:07 2024-09-15T02:33:36.333 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.5 MB 00:02 2024-09-15T02:33:37.792 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 19 kB 00:00 2024-09-15T02:33:38.646 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-15T02:33:40.456 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.5 MB 00:00 2024-09-15T02:33:42.156 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 19 kB 00:00 2024-09-15T02:33:43.027 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-15T02:33:44.126 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-15T02:33:46.149 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T02:33:46.149 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T02:33:46.260 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:33:46.265 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:33:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:33:46.267 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-09-15T02:33:46.267 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:33:46.267 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 79 M 2024-09-15T02:33:46.267 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 126 M 2024-09-15T02:33:46.268 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:33:46.611 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 76 kB/s | 23 kB 00:00 2024-09-15T02:33:48.195 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 8.3 MB/s | 16 MB 00:01 2024-09-15T02:33:48.636 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 14 MB/s | 29 MB 00:02 2024-09-15T02:33:48.733 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T02:33:49.120 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 12 MB/s | 35 MB 00:02 2024-09-15T02:33:49.120 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:33:49.120 INFO:teuthology.orchestra.run.smithi104.stdout:Total 28 MB/s | 79 MB 00:02 2024-09-15T02:33:49.800 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:33:50.154 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:33:50.154 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:33:50.831 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T02:33:50.831 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T02:33:50.939 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:33:50.944 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:33:50.944 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-09-15T02:33:50.944 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:33:50.944 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:33:50.944 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T02:33:50.944 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:33:50.944 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T02:33:50.945 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T02:33:50.945 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T02:33:50.945 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:33:50.945 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:33:50.945 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:33:50.945 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-09-15T02:33:50.945 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:33:50.946 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 79 M 2024-09-15T02:33:50.946 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 126 M 2024-09-15T02:33:50.946 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:33:51.112 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:33:51.113 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:33:51.193 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 189 kB/s | 23 kB 00:00 2024-09-15T02:33:51.785 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-15T02:33:52.352 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-15T02:33:52.744 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 18 MB/s | 29 MB 00:01 2024-09-15T02:33:52.744 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:33:52.744 INFO:teuthology.orchestra.run.smithi053.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-15T02:33:53.456 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:33:53.545 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:33:53.854 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:33:53.855 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:33:54.193 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T02:33:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:33:54.814 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:33:54.814 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:33:57.081 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:33:57.221 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:33:57.394 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:33:58.087 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T02:33:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:34:00.983 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:34:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:34:02.052 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:34:02.171 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:06.253 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:34:06.300 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:07.009 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:11.440 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:45.203 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:45.241 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:45.532 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:45.532 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T02:34:45.533 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:34:45.533 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:34:46.363 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:34:46.364 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:34:46.731 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T02:34:46.813 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T02:34:46.814 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T02:34:46.814 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T02:34:46.814 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:34:46.814 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:34:46.814 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T02:34:46.814 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:34:46.815 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T02:34:46.872 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T02:34:46.873 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T02:34:46.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-09-15T02:34:46.904 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 82.el9 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8274670 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T02:34:46.905 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:34:46.906 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T02:34:46.907 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T02:34:46.907 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 5.14.0-505.el9.x86_64 2024-09-15T02:34:46.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T02:34:47.610 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-09-15T02:34:48.909 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T02:34:48.937 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-09-15T02:34:48.940 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-15T02:34:48.955 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.xTQa4eFKEZ 2024-09-15T02:34:48.955 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xTQa4eFKEZ 2024-09-15T02:34:49.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.xTQa4eFKEZ 2024-09-15T02:34:49.155 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.xTQa4eFKEZ is 6KB 2024-09-15T02:34:49.203 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.xTQa4eFKEZ 2024-09-15T02:34:49.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T02:34:49.280 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T02:34:49.281 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T02:34:49.281 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T02:34:49.281 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T02:34:49.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T02:34:49.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-09-15T02:34:50.796 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:50.837 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T02:34:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:34:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:34:52.087 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:34:52.087 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:34:52.087 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:34:52.087 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T02:34:52.087 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T02:34:52.088 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T02:34:52.088 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T02:34:52.088 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:34:52.088 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:34:52.422 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T02:34:52.511 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T02:34:52.512 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T02:34:52.512 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T02:34:52.512 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:34:52.512 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:34:52.512 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T02:34:52.512 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:34:52.512 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T02:34:52.571 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T02:34:52.571 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T02:34:52.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 82.el9 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8274670 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T02:34:52.603 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:34:52.604 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T02:34:52.605 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T02:34:52.605 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 5.14.0-505.el9.x86_64 2024-09-15T02:34:52.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T02:34:53.245 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-09-15T02:34:54.579 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T02:34:54.622 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-09-15T02:34:54.624 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-09-15T02:34:54.639 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.x1pahm6JTQ 2024-09-15T02:34:54.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.x1pahm6JTQ 2024-09-15T02:34:54.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.x1pahm6JTQ 2024-09-15T02:34:54.859 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.x1pahm6JTQ is 6KB 2024-09-15T02:34:54.907 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.x1pahm6JTQ 2024-09-15T02:34:54.922 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T02:34:54.984 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T02:34:54.984 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T02:34:54.985 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T02:34:54.985 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T02:34:54.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T02:34:55.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-09-15T02:35:19.461 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T02:35:19.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T02:35:19.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:35:25.125 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T02:35:25.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-15T02:35:25.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:35:37.917 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-15T02:35:43.549 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-09-15T02:35:46.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T02:35:46.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:35:49.981 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-15T02:35:52.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-15T02:35:52.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:36:01.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T02:36:01.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:36:10.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-09-15T02:36:20.417 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-15T02:36:22.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-15T02:36:22.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:36:23.489 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-09-15T02:36:23.970 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-15T02:36:23.970 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T02:36:23.970 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-09-15T02:36:23.986 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-505.el9.x86_64 2024-09-15T02:36:23.986 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T02:36:23.986 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:36:23.986 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T02:36:24.987 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T02:36:24.988 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:36:25.063 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-09-15T02:36:25.084 DEBUG:teuthology.parallel:result is None 2024-09-15T02:36:35.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T02:36:35.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:36:35.790 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-15T02:36:36.180 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T02:36:36.180 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-15T02:36:36.180 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-15T02:36:36.194 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-505.el9.x86_64 2024-09-15T02:36:36.195 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T02:36:36.195 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:36:36.195 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T02:36:37.196 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T02:36:37.196 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:36:37.317 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-09-15T02:36:37.344 DEBUG:teuthology.parallel:result is None 2024-09-15T02:36:37.344 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T02:36:37.352 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T02:36:37.352 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:36:37.355 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:36:37.402 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T02:36:37.408 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T02:36:37.440 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T02:36:37.440 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:36:37.442 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:36:37.498 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T02:36:37.505 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T02:36:37.505 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:36:37.523 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:36:37.558 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:36:37.571 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:36:37.597 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:36:37.607 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:36:37.609 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T02:36:37.616 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T02:36:37.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:36:37.618 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:36:37.645 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T02:36:37.654 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T02:36:37.654 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:36:37.687 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:36:37.703 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:36:37.789 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:36:37.837 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:36:37.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:36:37.963 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:36:37.963 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:36:38.029 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:36:38.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:36:38.215 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:36:38.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:36:38.298 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:36:38.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:36:38.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-09-15T02:36:38.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-09-15T02:36:38.421 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T02:36:38.451 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T02:36:38.722 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T02:36:38.729 INFO:teuthology.task.internal:Starting timer... 2024-09-15T02:36:38.729 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T02:36:38.760 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T02:36:38.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-09-15T02:36:38.857 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-09-15T02:36:38.858 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-09-15T02:36:38.917 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-09-15T02:36:38.919 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T02:36:38.920 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-09-15T02:36:38.955 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-09-15T02:36:38.955 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-09-15T02:36:38.991 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-09-15T02:36:38.991 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-09-15T02:36:38.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T02:36:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:36:39.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T02:36:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:36:39.057 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T02:36:39.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-09-15T02:36:39.099 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-09-15T02:36:39.128 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T02:36:39.139 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T02:36:39.150 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T02:36:39.151 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T02:40:26.345 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-09-15T02:40:26.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-15T02:40:26.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:40:26.424 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-09-15T02:40:26.505 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-15T02:40:26.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T02:40:26.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:40:26.581 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-15T02:40:26.660 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T02:40:26.660 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T02:40:26.670 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T02:40:26.670 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:40:26.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:40:26.673 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:40:26.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:40:26.701 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T02:40:26.716 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T02:40:26.729 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T02:40:26.742 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-09-15T02:40:26.743 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T02:40:26.753 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-09-15T02:40:26.768 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-09-15T02:40:26.768 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T02:40:26.779 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-09-15T02:40:26.783 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T02:40:26.793 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T02:40:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T02:40:26.845 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:40:26.849 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:40:26.849 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-09-15T02:40:26.849 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.849 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.849 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.849 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.873 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:40:26.877 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:40:26.877 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-09-15T02:40:26.877 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.877 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.877 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.877 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:40:26.878 INFO:teuthology.run_tasks:Running task install... 2024-09-15T02:40:26.886 DEBUG:teuthology.task.install:project ceph 2024-09-15T02:40:26.886 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T02:40:26.886 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T02:40:26.886 INFO:teuthology.task.install:Using flavor: default 2024-09-15T02:40:26.891 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T02:40:26.891 INFO:teuthology.task.install:extra packages: [] 2024-09-15T02:40:26.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T02:40:26.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T02:40:27.472 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:44 ago on Sun 15 Sep 2024 02:39:43 AM UTC. 2024-09-15T02:40:27.480 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:50 ago on Sun 15 Sep 2024 02:39:37 AM UTC. 2024-09-15T02:40:27.562 INFO:teuthology.orchestra.run.smithi053.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T02:40:27.568 INFO:teuthology.orchestra.run.smithi104.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T02:40:27.606 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:40:27.607 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:40:27.607 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:40:27.610 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:40:27.611 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:40:27.612 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:40:27.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo dnf -y copr enable ceph/el9 2024-09-15T02:40:27.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo dnf -y copr enable ceph/el9 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:of the main distribution, and quality may vary. 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:, 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:and packages are not held to any quality or security level. 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T02:40:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T02:40:27.849 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T02:40:27.849 INFO:teuthology.orchestra.run.smithi053.stderr:of the main distribution, and quality may vary. 2024-09-15T02:40:27.849 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:40:27.849 INFO:teuthology.orchestra.run.smithi053.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T02:40:27.849 INFO:teuthology.orchestra.run.smithi053.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T02:40:27.849 INFO:teuthology.orchestra.run.smithi053.stderr:, 2024-09-15T02:40:27.849 INFO:teuthology.orchestra.run.smithi053.stderr:and packages are not held to any quality or security level. 2024-09-15T02:40:27.850 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:40:27.850 INFO:teuthology.orchestra.run.smithi053.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T02:40:27.850 INFO:teuthology.orchestra.run.smithi053.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T02:40:27.914 INFO:teuthology.orchestra.run.smithi104.stdout:Repository successfully enabled. 2024-09-15T02:40:27.921 INFO:teuthology.orchestra.run.smithi053.stdout:Repository successfully enabled. 2024-09-15T02:40:27.952 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T02:40:27.952 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:40:27.959 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T02:40:27.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:40:28.102 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T02:40:28.102 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T02:40:28.117 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T02:40:28.117 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T02:40:28.211 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T02:40:28.212 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:40:28.212 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T02:40:28.224 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T02:40:28.224 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:40:28.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T02:40:28.238 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T02:40:28.238 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T02:40:28.250 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T02:40:28.250 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T02:40:28.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T02:40:28.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T02:40:28.377 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T02:40:28.394 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T02:40:28.442 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-09-15T02:40:28.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-09-15T02:40:28.459 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-09-15T02:40:28.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-09-15T02:40:28.679 INFO:teuthology.orchestra.run.smithi104.stdout:45 files removed 2024-09-15T02:40:28.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2024-09-15T02:40:28.711 INFO:teuthology.orchestra.run.smithi053.stdout:45 files removed 2024-09-15T02:40:28.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw 2024-09-15T02:40:29.267 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 287 kB/s | 77 kB 00:00 2024-09-15T02:40:29.317 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 279 kB/s | 77 kB 00:00 2024-09-15T02:40:29.551 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T02:40:29.601 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T02:40:29.826 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T02:40:29.877 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T02:40:30.019 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for el9 owned by ceph 49 kB/s | 7.6 kB 00:00 2024-09-15T02:40:30.044 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for el9 owned by ceph 59 kB/s | 7.6 kB 00:00 2024-09-15T02:40:31.344 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.2 MB 00:01 2024-09-15T02:40:32.052 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.2 MB 00:01 2024-09-15T02:40:33.320 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-15T02:40:35.161 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-09-15T02:40:37.888 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-15T02:40:39.420 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-15T02:40:39.572 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-15T02:40:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T02:40:40.846 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 106 kB/s | 19 kB 00:00 2024-09-15T02:40:41.950 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T02:40:46.282 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T02:40:47.630 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 5.4 kB/s | 1.7 kB 00:00 2024-09-15T02:40:48.843 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T02:40:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T02:40:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout:Install 22 Packages 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2024-09-15T02:40:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:40:48.849 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 58 M 2024-09-15T02:40:48.849 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:40:49.958 INFO:teuthology.orchestra.run.smithi053.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 6.9 MB/s | 5.0 MB 00:00 2024-09-15T02:40:50.000 INFO:teuthology.orchestra.run.smithi053.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 607 kB/s | 25 kB 00:00 2024-09-15T02:40:50.084 INFO:teuthology.orchestra.run.smithi053.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-15T02:40:50.164 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:40:50.167 INFO:teuthology.orchestra.run.smithi053.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 5.4 MB/s | 458 kB 00:00 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T02:40:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T02:40:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout:Install 22 Packages 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2024-09-15T02:40:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:40:50.185 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 58 M 2024-09-15T02:40:50.185 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:40:50.617 INFO:teuthology.orchestra.run.smithi053.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 10 MB/s | 14 MB 00:01 2024-09-15T02:40:50.692 INFO:teuthology.orchestra.run.smithi053.stdout:(6/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 9.4 MB/s | 4.9 MB 00:00 2024-09-15T02:40:50.720 INFO:teuthology.orchestra.run.smithi053.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 451 kB/s | 45 kB 00:00 2024-09-15T02:40:50.743 INFO:teuthology.orchestra.run.smithi053.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.3 MB/s | 120 kB 00:00 2024-09-15T02:40:50.769 INFO:teuthology.orchestra.run.smithi053.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 3.0 MB/s | 156 kB 00:00 2024-09-15T02:40:50.794 INFO:teuthology.orchestra.run.smithi053.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 6.2 MB/s | 320 kB 00:00 2024-09-15T02:40:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 5.8 MB/s | 295 kB 00:00 2024-09-15T02:40:50.844 INFO:teuthology.orchestra.run.smithi053.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T02:40:51.120 INFO:teuthology.orchestra.run.smithi053.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 12 MB/s | 22 MB 00:01 2024-09-15T02:40:51.136 INFO:teuthology.orchestra.run.smithi053.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 155 kB/s | 45 kB 00:00 2024-09-15T02:40:51.178 INFO:teuthology.orchestra.run.smithi053.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-15T02:40:51.181 INFO:teuthology.orchestra.run.smithi104.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 6.9 MB/s | 5.0 MB 00:00 2024-09-15T02:40:51.212 INFO:teuthology.orchestra.run.smithi053.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 42 kB 00:00 2024-09-15T02:40:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 759 kB/s | 25 kB 00:00 2024-09-15T02:40:51.237 INFO:teuthology.orchestra.run.smithi053.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-15T02:40:51.290 INFO:teuthology.orchestra.run.smithi104.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 8.3 MB/s | 637 kB 00:00 2024-09-15T02:40:51.296 INFO:teuthology.orchestra.run.smithi053.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-15T02:40:51.321 INFO:teuthology.orchestra.run.smithi053.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-09-15T02:40:51.346 INFO:teuthology.orchestra.run.smithi053.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2024-09-15T02:40:51.366 INFO:teuthology.orchestra.run.smithi104.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 6.0 MB/s | 458 kB 00:00 2024-09-15T02:40:51.372 INFO:teuthology.orchestra.run.smithi053.stdout:(21/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2024-09-15T02:40:51.439 INFO:teuthology.orchestra.run.smithi053.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-15T02:40:51.706 INFO:teuthology.orchestra.run.smithi053.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 8.6 MB/s | 3.1 MB 00:00 2024-09-15T02:40:51.822 INFO:teuthology.orchestra.run.smithi053.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 6.5 MB/s | 2.9 MB 00:00 2024-09-15T02:40:51.828 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:40:51.828 INFO:teuthology.orchestra.run.smithi053.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-15T02:40:51.895 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:40:51.907 INFO:teuthology.orchestra.run.smithi104.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 10 MB/s | 14 MB 00:01 2024-09-15T02:40:51.933 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:40:51.933 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:40:51.983 INFO:teuthology.orchestra.run.smithi104.stdout:(6/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 8.0 MB/s | 4.9 MB 00:00 2024-09-15T02:40:51.999 INFO:teuthology.orchestra.run.smithi104.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 492 kB/s | 45 kB 00:00 2024-09-15T02:40:52.033 INFO:teuthology.orchestra.run.smithi104.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.3 MB/s | 120 kB 00:00 2024-09-15T02:40:52.058 INFO:teuthology.orchestra.run.smithi104.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.6 MB/s | 156 kB 00:00 2024-09-15T02:40:52.083 INFO:teuthology.orchestra.run.smithi104.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 6.2 MB/s | 320 kB 00:00 2024-09-15T02:40:52.109 INFO:teuthology.orchestra.run.smithi104.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 5.7 MB/s | 295 kB 00:00 2024-09-15T02:40:52.134 INFO:teuthology.orchestra.run.smithi104.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T02:40:52.359 INFO:teuthology.orchestra.run.smithi104.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 12 MB/s | 22 MB 00:01 2024-09-15T02:40:52.410 INFO:teuthology.orchestra.run.smithi104.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 164 kB/s | 45 kB 00:00 2024-09-15T02:40:52.435 INFO:teuthology.orchestra.run.smithi104.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 492 kB/s | 160 kB 00:00 2024-09-15T02:40:52.477 INFO:teuthology.orchestra.run.smithi104.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-15T02:40:52.510 INFO:teuthology.orchestra.run.smithi104.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-15T02:40:52.552 INFO:teuthology.orchestra.run.smithi104.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-15T02:40:52.577 INFO:teuthology.orchestra.run.smithi104.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-09-15T02:40:52.601 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:40:52.602 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:40:52.603 INFO:teuthology.orchestra.run.smithi104.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 537 kB/s | 49 kB 00:00 2024-09-15T02:40:52.630 INFO:teuthology.orchestra.run.smithi104.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-09-15T02:40:52.680 INFO:teuthology.orchestra.run.smithi104.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-15T02:40:52.814 INFO:teuthology.orchestra.run.smithi104.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 16 MB/s | 2.9 MB 00:00 2024-09-15T02:40:52.872 INFO:teuthology.orchestra.run.smithi104.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 11 MB/s | 3.1 MB 00:00 2024-09-15T02:40:52.876 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:40:52.876 INFO:teuthology.orchestra.run.smithi104.stdout:Total 22 MB/s | 58 MB 00:02 2024-09-15T02:40:52.938 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:40:52.970 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:40:52.970 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:40:53.602 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:40:53.603 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:40:53.979 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:40:54.381 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T02:40:54.441 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:40:54.534 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:40:54.623 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:40:54.669 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:40:54.757 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:40:54.811 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T02:40:54.836 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:40:54.856 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T02:40:55.169 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T02:40:55.274 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:40:55.321 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T02:40:55.354 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:40:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:40:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:40:55.436 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:40:55.454 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T02:40:55.467 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:40:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T02:40:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:40:55.555 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:40:55.601 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T02:40:55.610 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:40:55.630 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T02:40:55.640 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:40:55.706 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:40:55.745 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T02:40:55.799 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T02:40:55.840 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T02:40:56.068 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T02:40:56.112 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:40:56.149 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T02:40:56.174 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:40:56.194 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:40:56.211 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T02:40:56.254 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T02:40:56.285 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:40:56.291 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:40:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:40:56.360 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T02:40:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:40:56.388 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T02:40:56.430 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:40:56.478 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T02:40:56.514 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T02:40:56.547 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T02:40:56.841 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T02:40:56.884 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:40:56.975 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:40:57.048 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T02:40:57.078 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T02:40:58.928 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:40:58.968 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:40:59.514 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:40:59.549 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:40:59.565 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:40:59.604 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:40:59.692 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:40:59.692 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T02:40:59.692 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T02:40:59.692 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:40:59.730 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:41:00.082 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:41:00.119 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:41:00.228 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:41:00.228 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T02:41:00.228 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T02:41:00.228 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:00.262 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:41:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:41:09.992 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-09-15T02:41:09.992 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-09-15T02:41:09.992 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-09-15T02:41:09.992 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-09-15T02:41:09.992 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-09-15T02:41:09.992 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-09-15T02:41:09.993 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-09-15T02:41:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:10.221 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:41:10.221 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-09-15T02:41:10.221 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-09-15T02:41:10.221 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-09-15T02:41:10.222 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-09-15T02:41:10.222 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-09-15T02:41:10.222 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-09-15T02:41:10.222 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-09-15T02:41:10.222 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:41:11.396 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:41:11.396 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:11.396 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T02:41:11.396 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:41:11.396 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:41:11.397 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:11.398 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:41:11.487 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:41:11.487 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:41:11.605 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:41:11.684 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:41:11.684 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:11.684 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T02:41:11.684 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:41:11.684 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:41:11.684 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:11.686 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:41:11.774 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:41:11.774 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T02:41:12.719 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T02:41:12.720 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T02:41:12.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T02:41:12.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T02:41:12.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T02:41:12.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T02:41:12.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:41:12.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T02:41:12.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T02:41:13.053 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T02:41:13.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T02:41:13.055 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T02:41:13.055 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T02:41:13.055 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T02:41:13.055 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:41:13.055 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T02:41:13.055 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.610 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T02:41:13.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:13.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.073 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.074 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.075 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T02:41:14.075 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.075 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.075 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:14.075 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T02:41:14.075 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:14.075 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:14.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test 2024-09-15T02:41:14.546 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:14.683 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:14.683 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:14.683 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:41:14.683 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:14.683 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:41:14.683 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-09-15T02:41:14.684 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:14.685 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 48 M 2024-09-15T02:41:14.685 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 193 M 2024-09-15T02:41:14.685 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:41:14.994 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:15.139 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T02:41:15.140 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:15.141 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:41:15.141 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:15.141 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-09-15T02:41:15.141 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:15.141 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 48 M 2024-09-15T02:41:15.141 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 193 M 2024-09-15T02:41:15.141 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:41:15.142 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-15T02:41:15.167 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 719 kB/s | 163 kB 00:00 2024-09-15T02:41:15.192 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-15T02:41:15.462 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 755 kB/s | 163 kB 00:00 2024-09-15T02:41:15.746 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 609 kB/s | 304 kB 00:00 2024-09-15T02:41:15.804 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-09-15T02:41:16.534 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 48 MB 00:01 2024-09-15T02:41:16.536 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:16.536 INFO:teuthology.orchestra.run.smithi104.stdout:Total 26 MB/s | 48 MB 00:01 2024-09-15T02:41:16.564 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:41:16.583 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:41:16.583 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:41:16.946 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:41:16.947 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:41:17.454 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 22 MB/s | 48 MB 00:02 2024-09-15T02:41:17.456 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:17.457 INFO:teuthology.orchestra.run.smithi053.stdout:Total 21 MB/s | 48 MB 00:02 2024-09-15T02:41:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:41:17.485 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:41:17.506 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:41:17.506 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:41:17.560 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T02:41:17.600 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T02:41:17.627 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:41:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:41:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:41:18.439 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:41:18.524 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T02:41:18.596 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T02:41:18.623 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:41:22.027 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:41:22.136 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:41:23.297 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:41:23.374 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:41:24.252 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:41:24.252 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T02:41:24.252 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T02:41:24.253 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T02:41:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T02:41:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:24.545 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:41:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T02:41:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T02:41:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:24.546 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:24.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2024-09-15T02:41:25.393 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:38 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:25.527 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T02:41:25.530 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T02:41:25.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T02:41:25.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout:Install 34 Packages 2024-09-15T02:41:25.533 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:25.534 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 27 M 2024-09-15T02:41:25.534 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 96 M 2024-09-15T02:41:25.534 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:41:25.589 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:41:25.589 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T02:41:25.589 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T02:41:25.590 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T02:41:25.857 INFO:teuthology.orchestra.run.smithi104.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 54 kB/s | 6.5 kB 00:00 2024-09-15T02:41:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T02:41:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:25.913 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:41:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T02:41:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T02:41:25.914 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:25.914 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:26.058 INFO:teuthology.orchestra.run.smithi104.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-15T02:41:26.099 INFO:teuthology.orchestra.run.smithi104.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-15T02:41:26.191 INFO:teuthology.orchestra.run.smithi104.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.6 MB/s | 240 kB 00:00 2024-09-15T02:41:26.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph 2024-09-15T02:41:26.292 INFO:teuthology.orchestra.run.smithi104.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 9.0 MB/s | 3.9 MB 00:00 2024-09-15T02:41:26.367 INFO:teuthology.orchestra.run.smithi104.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-15T02:41:26.409 INFO:teuthology.orchestra.run.smithi104.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T02:41:26.568 INFO:teuthology.orchestra.run.smithi104.stdout:(8/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-15T02:41:26.593 INFO:teuthology.orchestra.run.smithi104.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 735 kB/s | 135 kB 00:00 2024-09-15T02:41:26.635 INFO:teuthology.orchestra.run.smithi104.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 5.0 MB/s | 215 kB 00:00 2024-09-15T02:41:26.669 INFO:teuthology.orchestra.run.smithi104.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 3.3 MB/s | 1.2 MB 00:00 2024-09-15T02:41:26.694 INFO:teuthology.orchestra.run.smithi104.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 993 kB/s | 125 kB 00:00 2024-09-15T02:41:26.736 INFO:teuthology.orchestra.run.smithi104.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 521 kB/s | 35 kB 00:00 2024-09-15T02:41:26.778 INFO:teuthology.orchestra.run.smithi104.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-15T02:41:26.793 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:40 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:26.803 INFO:teuthology.orchestra.run.smithi104.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 761 kB/s | 51 kB 00:00 2024-09-15T02:41:26.828 INFO:teuthology.orchestra.run.smithi104.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 575 kB/s | 77 kB 00:00 2024-09-15T02:41:26.854 INFO:teuthology.orchestra.run.smithi104.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-15T02:41:26.904 INFO:teuthology.orchestra.run.smithi104.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-15T02:41:26.929 INFO:teuthology.orchestra.run.smithi104.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 459 kB/s | 11 kB 00:00 2024-09-15T02:41:26.940 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:26.942 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:26.942 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T02:41:26.943 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T02:41:26.944 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T02:41:26.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout:Install 34 Packages 2024-09-15T02:41:26.946 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:26.947 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 27 M 2024-09-15T02:41:26.947 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 96 M 2024-09-15T02:41:26.947 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:41:26.954 INFO:teuthology.orchestra.run.smithi104.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-15T02:41:26.980 INFO:teuthology.orchestra.run.smithi104.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-15T02:41:27.009 INFO:teuthology.orchestra.run.smithi104.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-15T02:41:27.030 INFO:teuthology.orchestra.run.smithi104.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-09-15T02:41:27.055 INFO:teuthology.orchestra.run.smithi104.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-15T02:41:27.081 INFO:teuthology.orchestra.run.smithi104.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-15T02:41:27.106 INFO:teuthology.orchestra.run.smithi104.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T02:41:27.131 INFO:teuthology.orchestra.run.smithi104.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-15T02:41:27.156 INFO:teuthology.orchestra.run.smithi104.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-15T02:41:27.182 INFO:teuthology.orchestra.run.smithi104.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-15T02:41:27.207 INFO:teuthology.orchestra.run.smithi104.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-15T02:41:27.232 INFO:teuthology.orchestra.run.smithi104.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-15T02:41:27.258 INFO:teuthology.orchestra.run.smithi104.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-09-15T02:41:27.283 INFO:teuthology.orchestra.run.smithi104.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T02:41:27.308 INFO:teuthology.orchestra.run.smithi104.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-15T02:41:27.312 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:27.312 INFO:teuthology.orchestra.run.smithi104.stdout:Total 15 MB/s | 27 MB 00:01 2024-09-15T02:41:27.435 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:41:27.471 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:41:27.471 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:41:27.662 INFO:teuthology.orchestra.run.smithi053.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-15T02:41:27.852 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:41:27.852 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:41:27.871 INFO:teuthology.orchestra.run.smithi053.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.3 MB/s | 2.1 MB 00:00 2024-09-15T02:41:27.906 INFO:teuthology.orchestra.run.smithi053.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 3.9 MB/s | 1.4 MB 00:00 2024-09-15T02:41:27.997 INFO:teuthology.orchestra.run.smithi053.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.6 MB/s | 240 kB 00:00 2024-09-15T02:41:28.189 INFO:teuthology.orchestra.run.smithi053.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 7.5 MB/s | 3.9 MB 00:00 2024-09-15T02:41:28.214 INFO:teuthology.orchestra.run.smithi053.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-15T02:41:28.390 INFO:teuthology.orchestra.run.smithi053.stdout:(7/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-15T02:41:28.415 INFO:teuthology.orchestra.run.smithi053.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-09-15T02:41:28.465 INFO:teuthology.orchestra.run.smithi053.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-15T02:41:28.549 INFO:teuthology.orchestra.run.smithi053.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-15T02:41:28.574 INFO:teuthology.orchestra.run.smithi053.stdout:(11/34): python3-pycparser-2.20-6.el9.noarch.rp 735 kB/s | 135 kB 00:00 2024-09-15T02:41:28.682 INFO:teuthology.orchestra.run.smithi053.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-09-15T02:41:28.732 INFO:teuthology.orchestra.run.smithi053.stdout:(13/34): python3-cryptography-36.0.1-4.el9.x86_ 2.3 MB/s | 1.2 MB 00:00 2024-09-15T02:41:28.761 INFO:teuthology.orchestra.run.smithi053.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 811 kB/s | 172 kB 00:00 2024-09-15T02:41:28.786 INFO:teuthology.orchestra.run.smithi053.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 743 kB/s | 77 kB 00:00 2024-09-15T02:41:28.828 INFO:teuthology.orchestra.run.smithi053.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 623 kB/s | 42 kB 00:00 2024-09-15T02:41:28.862 INFO:teuthology.orchestra.run.smithi053.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 577 kB/s | 43 kB 00:00 2024-09-15T02:41:28.887 INFO:teuthology.orchestra.run.smithi053.stdout:(18/34): python3-pytz-2021.1-5.el9.noarch.rpm 330 kB/s | 51 kB 00:00 2024-09-15T02:41:28.921 INFO:teuthology.orchestra.run.smithi053.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-09-15T02:41:28.946 INFO:teuthology.orchestra.run.smithi053.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-09-15T02:41:28.971 INFO:teuthology.orchestra.run.smithi053.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 366 kB/s | 18 kB 00:00 2024-09-15T02:41:29.038 INFO:teuthology.orchestra.run.smithi053.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 348 kB/s | 23 kB 00:00 2024-09-15T02:41:29.063 INFO:teuthology.orchestra.run.smithi053.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-09-15T02:41:29.089 INFO:teuthology.orchestra.run.smithi053.stdout:(24/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-15T02:41:29.101 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:41:29.114 INFO:teuthology.orchestra.run.smithi053.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-15T02:41:29.139 INFO:teuthology.orchestra.run.smithi053.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-15T02:41:29.165 INFO:teuthology.orchestra.run.smithi053.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-15T02:41:29.169 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T02:41:29.198 INFO:teuthology.orchestra.run.smithi053.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-09-15T02:41:29.201 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T02:41:29.223 INFO:teuthology.orchestra.run.smithi053.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-15T02:41:29.234 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T02:41:29.249 INFO:teuthology.orchestra.run.smithi053.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-09-15T02:41:29.274 INFO:teuthology.orchestra.run.smithi053.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-15T02:41:29.299 INFO:teuthology.orchestra.run.smithi053.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-15T02:41:29.324 INFO:teuthology.orchestra.run.smithi053.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T02:41:29.326 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T02:41:29.350 INFO:teuthology.orchestra.run.smithi053.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-15T02:41:29.354 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:29.355 INFO:teuthology.orchestra.run.smithi053.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-15T02:41:29.422 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T02:41:29.462 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T02:41:29.490 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:41:29.522 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T02:41:29.532 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:41:29.532 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:41:29.552 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T02:41:29.586 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T02:41:29.619 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T02:41:29.660 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T02:41:29.707 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T02:41:29.741 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T02:41:29.789 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T02:41:29.821 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:41:29.894 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T02:41:29.952 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:41:29.952 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:41:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T02:41:30.120 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T02:41:30.182 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T02:41:30.221 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T02:41:30.270 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T02:41:30.325 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T02:41:30.512 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T02:41:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T02:41:30.645 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T02:41:30.741 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T02:41:30.773 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T02:41:30.889 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T02:41:31.036 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T02:41:31.068 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:41:31.093 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:41:31.093 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:31.094 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T02:41:31.094 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:41:31.094 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:41:31.094 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:31.211 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:41:31.286 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T02:41:31.319 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T02:41:31.361 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T02:41:31.466 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T02:41:31.559 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T02:41:31.605 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T02:41:31.648 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T02:41:31.694 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T02:41:31.729 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T02:41:31.762 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T02:41:31.803 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T02:41:31.847 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T02:41:31.880 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T02:41:31.930 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T02:41:31.970 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:41:32.047 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T02:41:32.200 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T02:41:32.296 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T02:41:32.347 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T02:41:32.402 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T02:41:32.460 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T02:41:32.477 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:41:32.507 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:41:32.507 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:32.507 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T02:41:32.507 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:41:32.507 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:41:32.508 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:32.525 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T02:41:32.730 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T02:41:32.781 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T02:41:32.861 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T02:41:32.903 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:41:32.927 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:41:32.928 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:32.928 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T02:41:32.928 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:41:32.928 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:41:32.928 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:32.961 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T02:41:32.996 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T02:41:33.120 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T02:41:33.136 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:41:33.160 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:41:33.160 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:33.160 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T02:41:33.160 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:41:33.160 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:41:33.160 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:33.224 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:41:33.281 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T02:41:33.314 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:41:33.342 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:41:33.342 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:33.342 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T02:41:33.342 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:41:33.342 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:41:33.342 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:34.732 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:41:34.760 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:41:34.760 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:34.761 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T02:41:34.761 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:41:34.761 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:41:34.761 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:41:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T02:41:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T02:41:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T02:41:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T02:41:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T02:41:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T02:41:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T02:41:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T02:41:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T02:41:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T02:41:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T02:41:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T02:41:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T02:41:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T02:41:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:41:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T02:41:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T02:41:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T02:41:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T02:41:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T02:41:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T02:41:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T02:41:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T02:41:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T02:41:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T02:41:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T02:41:35.157 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:41:35.183 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:41:35.183 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:35.183 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T02:41:35.183 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:41:35.183 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:41:35.183 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:35.391 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:41:35.416 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:41:35.417 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:35.417 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T02:41:35.417 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:41:35.417 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:41:35.417 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:35.482 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T02:41:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T02:41:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T02:41:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T02:41:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T02:41:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T02:41:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T02:41:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:35.640 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:36.000 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2024-09-15T02:41:36.567 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:49 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:36.658 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:41:36.699 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:36.700 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:41:36.700 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:36.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2024-09-15T02:41:37.262 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T02:41:37.263 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T02:41:37.264 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T02:41:37.265 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T02:41:37.265 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T02:41:37.265 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T02:41:37.266 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T02:41:37.266 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T02:41:37.266 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T02:41:37.266 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T02:41:37.266 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T02:41:37.330 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:50 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:37.463 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:37.463 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:37.463 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:41:37.463 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:37.463 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:41:37.463 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 83 k 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 375 k 2024-09-15T02:41:37.464 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:41:37.658 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 426 kB/s | 83 kB 00:00 2024-09-15T02:41:37.659 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:37.659 INFO:teuthology.orchestra.run.smithi104.stdout:Total 422 kB/s | 83 kB 00:00 2024-09-15T02:41:37.659 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:41:37.665 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:41:37.665 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:41:37.666 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:41:37.667 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:41:37.741 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:41:38.191 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:38.326 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T02:41:38.326 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:38.326 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T02:41:38.327 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T02:41:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:38.329 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:38.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base 2024-09-15T02:41:38.866 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:38.866 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:38.866 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:41:38.866 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:38.866 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:38.866 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:39.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T02:41:39.281 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:39.374 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:41:39.418 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:39.419 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:41:39.419 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:39.501 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install cephadm 2024-09-15T02:41:39.638 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:52 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:39.775 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:39.775 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-15T02:41:39.775 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:39.775 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:41:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T02:41:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:39.776 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:41:39.776 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:39.776 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:41:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:39.776 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 137 k 2024-09-15T02:41:39.776 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 379 k 2024-09-15T02:41:39.777 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:41:39.989 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 649 kB/s | 137 kB 00:00 2024-09-15T02:41:39.989 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:39.989 INFO:teuthology.orchestra.run.smithi104.stdout:Total 644 kB/s | 137 kB 00:00 2024-09-15T02:41:39.990 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:41:39.997 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:41:39.997 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:41:40.055 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:41:40.055 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:41:40.089 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:40.196 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:41:40.226 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:41:40.227 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:40.228 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 83 k 2024-09-15T02:41:40.228 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 375 k 2024-09-15T02:41:40.228 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:41:40.256 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:41:40.281 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:41:40.281 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:40.281 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T02:41:40.281 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:40.422 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 426 kB/s | 83 kB 00:00 2024-09-15T02:41:40.423 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:40.423 INFO:teuthology.orchestra.run.smithi053.stdout:Total 423 kB/s | 83 kB 00:00 2024-09-15T02:41:40.423 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:41:40.429 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:41:40.429 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:41:40.431 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:41:40.431 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:41:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:41:40.950 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:41.054 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:41.269 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:41:41.269 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:41.269 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:41:41.269 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:41.269 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:41.269 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:41.394 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:41.517 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2024-09-15T02:41:41.636 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:41:41.636 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:41.636 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:41:41.636 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:41.636 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:41.636 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:41.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T02:41:42.092 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:42.199 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:41:42.240 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:42.241 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:41:42.241 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:42.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T02:41:42.450 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:42.588 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:41:42.589 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:42.590 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 137 k 2024-09-15T02:41:42.590 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 379 k 2024-09-15T02:41:42.590 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:41:42.796 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 668 kB/s | 137 kB 00:00 2024-09-15T02:41:42.796 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:42.797 INFO:teuthology.orchestra.run.smithi053.stdout:Total 663 kB/s | 137 kB 00:00 2024-09-15T02:41:42.797 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:41:42.805 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:41:42.806 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:41:42.866 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:41:42.866 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:41:42.872 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:43.007 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:43.007 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:41:43.007 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:41:43.008 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:43.009 INFO:teuthology.orchestra.run.smithi104.stdout:Install 7 Packages 2024-09-15T02:41:43.009 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:43.009 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.7 M 2024-09-15T02:41:43.009 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 80 M 2024-09-15T02:41:43.009 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:41:43.018 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:41:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:41:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:41:43.133 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:41:43.133 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T02:41:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:43.191 INFO:teuthology.orchestra.run.smithi104.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 97 kB/s | 15 kB 00:00 2024-09-15T02:41:43.216 INFO:teuthology.orchestra.run.smithi104.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1643.gd75 139 kB/s | 24 kB 00:00 2024-09-15T02:41:43.258 INFO:teuthology.orchestra.run.smithi104.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-09-15T02:41:43.283 INFO:teuthology.orchestra.run.smithi104.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T02:41:43.309 INFO:teuthology.orchestra.run.smithi104.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-15T02:41:43.334 INFO:teuthology.orchestra.run.smithi104.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T02:41:43.434 INFO:teuthology.orchestra.run.smithi104.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.5 MB/s | 3.4 MB 00:00 2024-09-15T02:41:43.437 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:43.438 INFO:teuthology.orchestra.run.smithi104.stdout:Total 8.5 MB/s | 3.7 MB 00:00 2024-09-15T02:41:43.438 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:41:43.446 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:41:43.446 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:41:43.529 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:41:43.529 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:41:43.835 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:41:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T02:41:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T02:41:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T02:41:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T02:41:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T02:41:44.054 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:41:44.054 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:44.055 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:41:44.055 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:41:44.055 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:44.055 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:44.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr 2024-09-15T02:41:44.921 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:45.015 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:41:45.041 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T02:41:45.060 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:45.061 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:41:45.061 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:41:45.141 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T02:41:45.724 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:59 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:45.863 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:45.864 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:45.864 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:41:45.864 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:45.864 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:41:45.865 INFO:teuthology.orchestra.run.smithi053.stdout:Install 7 Packages 2024-09-15T02:41:45.866 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:45.866 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.7 M 2024-09-15T02:41:45.866 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 80 M 2024-09-15T02:41:45.866 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:41:46.068 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:41:46.068 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T02:41:46.068 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T02:41:46.068 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T02:41:46.069 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T02:41:46.069 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T02:41:46.069 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T02:41:46.074 INFO:teuthology.orchestra.run.smithi053.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 168 kB/s | 24 kB 00:00 2024-09-15T02:41:46.091 INFO:teuthology.orchestra.run.smithi053.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 90 kB/s | 15 kB 00:00 2024-09-15T02:41:46.124 INFO:teuthology.orchestra.run.smithi053.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 181 kB/s | 9.0 kB 00:00 2024-09-15T02:41:46.141 INFO:teuthology.orchestra.run.smithi053.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-15T02:41:46.158 INFO:teuthology.orchestra.run.smithi053.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 608 kB/s | 41 kB 00:00 2024-09-15T02:41:46.183 INFO:teuthology.orchestra.run.smithi053.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-15T02:41:46.284 INFO:teuthology.orchestra.run.smithi053.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 9.4 MB/s | 3.4 MB 00:00 2024-09-15T02:41:46.287 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:46.287 INFO:teuthology.orchestra.run.smithi053.stdout:Total 8.7 MB/s | 3.7 MB 00:00 2024-09-15T02:41:46.287 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:41:46.296 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:41:46.296 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:41:46.352 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T02:41:46.352 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:46.352 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:41:46.352 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:46.353 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:46.380 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:41:46.380 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:41:46.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T02:41:46.693 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:41:46.752 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T02:41:46.786 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T02:41:46.842 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T02:41:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T02:41:46.918 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T02:41:47.131 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:00 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:47.264 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T02:41:47.265 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:41:47.266 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-09-15T02:41:47.267 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-09-15T02:41:47.267 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:47.267 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 38 M 2024-09-15T02:41:47.267 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 212 M 2024-09-15T02:41:47.267 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:41:47.599 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-15T02:41:47.666 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.1 MB/s | 801 kB 00:00 2024-09-15T02:41:47.691 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-09-15T02:41:47.742 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-15T02:41:47.851 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 17 MB/s | 7.4 MB 00:00 2024-09-15T02:41:47.878 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 284 kB/s | 38 kB 00:00 2024-09-15T02:41:47.968 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T02:41:47.995 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-15T02:41:48.013 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:41:48.201 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-09-15T02:41:48.253 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-09-15T02:41:48.279 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 3.2 MB/s | 250 kB 00:00 2024-09-15T02:41:48.345 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 10 MB/s | 4.9 MB 00:00 2024-09-15T02:41:48.881 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-09-15T02:41:48.881 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:48.881 INFO:teuthology.orchestra.run.smithi104.stdout:Total 24 MB/s | 38 MB 00:01 2024-09-15T02:41:49.012 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:41:49.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T02:41:49.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T02:41:49.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T02:41:49.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T02:41:49.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T02:41:49.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T02:41:49.235 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:41:49.261 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:41:49.261 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:41:49.295 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:49.296 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:41:49.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T02:41:49.772 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:41:49.772 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:41:50.245 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:04 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:41:50.384 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:41:50.385 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-09-15T02:41:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:41:50.385 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-09-15T02:41:50.385 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:41:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T02:41:50.385 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:41:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:50.386 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:41:50.387 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-09-15T02:41:50.387 INFO:teuthology.orchestra.run.smithi053.stdout:Install 12 Packages 2024-09-15T02:41:50.387 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:41:50.387 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 38 M 2024-09-15T02:41:50.387 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 212 M 2024-09-15T02:41:50.387 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:41:50.526 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:41:50.586 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T02:41:50.635 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:41:50.734 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:41:50.770 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T02:41:50.791 INFO:teuthology.orchestra.run.smithi053.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-15T02:41:50.908 INFO:teuthology.orchestra.run.smithi053.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-15T02:41:50.933 INFO:teuthology.orchestra.run.smithi053.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-09-15T02:41:50.954 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T02:41:50.988 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T02:41:50.992 INFO:teuthology.orchestra.run.smithi053.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-09-15T02:41:51.069 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T02:41:51.092 INFO:teuthology.orchestra.run.smithi053.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-15T02:41:51.118 INFO:teuthology.orchestra.run.smithi053.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 308 kB/s | 38 kB 00:00 2024-09-15T02:41:51.226 INFO:teuthology.orchestra.run.smithi053.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-15T02:41:51.352 INFO:teuthology.orchestra.run.smithi053.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.7 MB/s | 3.0 MB 00:00 2024-09-15T02:41:51.418 INFO:teuthology.orchestra.run.smithi053.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-09-15T02:41:51.494 INFO:teuthology.orchestra.run.smithi053.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-09-15T02:41:51.577 INFO:teuthology.orchestra.run.smithi053.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 10 MB/s | 4.9 MB 00:00 2024-09-15T02:41:51.706 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T02:41:51.823 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T02:41:51.903 INFO:teuthology.orchestra.run.smithi053.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-15T02:41:51.903 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:51.904 INFO:teuthology.orchestra.run.smithi053.stdout:Total 25 MB/s | 38 MB 00:01 2024-09-15T02:41:52.266 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:41:52.293 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:41:52.293 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:41:52.817 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:41:52.817 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:41:53.071 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:41:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:41:53.667 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T02:41:53.708 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:41:53.813 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:41:53.861 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T02:41:54.070 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T02:41:54.093 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T02:41:54.101 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T02:41:54.125 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:41:54.189 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T02:41:54.870 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T02:41:54.984 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T02:41:56.322 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:41:56.665 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:41:56.666 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T02:41:57.100 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T02:41:57.101 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T02:41:57.102 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T02:41:57.102 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T02:41:57.102 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T02:41:57.102 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T02:41:57.102 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:57.102 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:41:57.401 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T02:41:57.424 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:41:57.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2024-09-15T02:41:58.020 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:11 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:41:58.157 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:41:58.157 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:58.157 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:41:58.157 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:58.157 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T02:41:58.158 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.2 M 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 26 M 2024-09-15T02:41:58.159 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:41:58.649 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 320 kB/s | 26 kB 00:00 2024-09-15T02:41:58.674 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 182 kB/s | 19 kB 00:00 2024-09-15T02:41:58.749 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-15T02:41:58.774 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-15T02:41:58.799 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 206 kB/s | 47 kB 00:00 2024-09-15T02:41:58.825 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-15T02:41:58.850 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-15T02:41:58.900 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-15T02:41:58.925 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-15T02:41:58.951 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T02:41:58.976 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-15T02:41:59.001 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-15T02:41:59.035 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-15T02:41:59.037 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:41:59.037 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-15T02:41:59.102 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:41:59.116 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:41:59.117 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:41:59.203 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:41:59.204 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:41:59.726 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:41:59.912 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T02:41:59.969 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T02:42:00.024 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T02:42:00.053 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T02:42:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:42:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T02:42:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:42:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:42:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T02:42:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T02:42:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T02:42:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T02:42:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T02:42:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T02:42:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:42:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T02:42:00.094 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T02:42:00.195 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T02:42:00.305 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T02:42:00.354 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T02:42:00.554 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:00.555 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:00.919 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-rook 2024-09-15T02:42:01.060 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T02:42:01.110 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T02:42:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T02:42:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T02:42:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:42:01.512 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:15 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:01.654 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:01.655 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:01.655 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:42:01.655 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:01.655 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:01.655 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T02:42:01.655 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T02:42:01.656 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T02:42:01.657 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:01.657 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:01.657 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:01.657 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-09-15T02:42:01.657 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:01.657 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.2 M 2024-09-15T02:42:01.658 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 26 M 2024-09-15T02:42:01.658 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:01.953 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T02:42:01.954 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T02:42:01.955 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T02:42:01.955 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T02:42:01.955 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T02:42:02.108 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 217 kB/s | 19 kB 00:00 2024-09-15T02:42:02.133 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 226 kB/s | 26 kB 00:00 2024-09-15T02:42:02.191 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 276 kB/s | 47 kB 00:00 2024-09-15T02:42:02.242 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-15T02:42:02.267 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-15T02:42:02.292 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T02:42:02.292 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:02.293 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:02.293 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:42:02.293 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T02:42:02.293 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T02:42:02.293 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T02:42:02.294 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T02:42:02.294 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T02:42:02.294 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T02:42:02.294 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T02:42:02.294 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T02:42:02.295 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T02:42:02.295 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T02:42:02.295 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T02:42:02.295 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T02:42:02.295 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:02.295 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:02.296 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-15T02:42:02.334 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-15T02:42:02.359 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-09-15T02:42:02.384 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 154 kB/s | 14 kB 00:00 2024-09-15T02:42:02.426 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-15T02:42:02.452 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-09-15T02:42:02.477 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-15T02:42:02.527 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-15T02:42:02.531 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:02.531 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-15T02:42:02.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T02:42:02.601 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:02.618 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:02.618 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:02.716 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:02.717 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:03.146 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:16 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:03.235 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:03.280 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:03.280 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:03.281 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:03.282 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:03.282 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-09-15T02:42:03.282 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:03.282 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 7.0 M 2024-09-15T02:42:03.282 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 33 M 2024-09-15T02:42:03.282 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:03.415 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T02:42:03.605 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T02:42:03.678 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T02:42:03.721 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T02:42:03.763 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T02:42:03.869 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T02:42:03.944 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 588 kB/s | 124 kB 00:00 2024-09-15T02:42:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T02:42:04.027 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-15T02:42:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T02:42:04.052 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-15T02:42:04.078 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-15T02:42:04.286 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 449 kB/s | 248 kB 00:00 2024-09-15T02:42:04.819 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T02:42:04.845 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.4 MB/s | 6.0 MB 00:01 2024-09-15T02:42:04.847 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:04.847 INFO:teuthology.orchestra.run.smithi104.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-09-15T02:42:04.861 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T02:42:04.895 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T02:42:04.916 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:04.927 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:04.927 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:04.942 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T02:42:04.964 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:42:05.025 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:05.025 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:05.324 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:05.484 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T02:42:05.530 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T02:42:05.838 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T02:42:05.839 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T02:42:05.839 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T02:42:06.048 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T02:42:06.140 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T02:42:06.176 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T02:42:06.176 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T02:42:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T02:42:06.178 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T02:42:06.178 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T02:42:06.178 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T02:42:06.178 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:06.178 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:06.195 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T02:42:06.225 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:42:06.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T02:42:06.969 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:42:06.970 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T02:42:06.970 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T02:42:06.970 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T02:42:06.970 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T02:42:06.970 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T02:42:07.028 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:07.167 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:07.168 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:07.168 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:42:07.168 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:07.168 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:07.169 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 7.0 M 2024-09-15T02:42:07.170 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 33 M 2024-09-15T02:42:07.170 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T02:42:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:07.233 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:42:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T02:42:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T02:42:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T02:42:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T02:42:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T02:42:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:07.234 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:07.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2024-09-15T02:42:07.570 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 586 kB/s | 124 kB 00:00 2024-09-15T02:42:07.596 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-15T02:42:07.646 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T02:42:07.671 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.3 MB/s | 548 kB 00:00 2024-09-15T02:42:07.696 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-15T02:42:08.052 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:08.187 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:08.187 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:08.187 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:42:08.187 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-09-15T02:42:08.188 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:08.189 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 893 k 2024-09-15T02:42:08.189 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 2.6 M 2024-09-15T02:42:08.189 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:08.387 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 449 kB/s | 80 kB 00:00 2024-09-15T02:42:08.496 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-15T02:42:08.496 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:08.497 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.8 MB/s | 893 kB 00:00 2024-09-15T02:42:08.508 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:08.530 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:08.530 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:08.611 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:08.611 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:08.772 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-09-15T02:42:08.774 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:08.774 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.4 MB/s | 7.0 MB 00:01 2024-09-15T02:42:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:08.848 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:08.860 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:08.860 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:08.947 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T02:42:08.966 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:08.967 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:09.012 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:42:09.411 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:09.586 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T02:42:09.625 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:42:09.625 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T02:42:09.641 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T02:42:09.888 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T02:42:09.888 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:09.888 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:09.888 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T02:42:09.888 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:09.889 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:10.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-volume 2024-09-15T02:42:10.202 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T02:42:10.296 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T02:42:10.366 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T02:42:10.394 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:42:10.761 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:10.895 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:10.896 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:10.897 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:42:10.897 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:10.897 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 270 k 2024-09-15T02:42:10.897 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 1.3 M 2024-09-15T02:42:10.897 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:11.109 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T02:42:11.109 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:11.110 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-15T02:42:11.110 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:11.118 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:11.118 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:11.149 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:11.149 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:11.184 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:42:11.184 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T02:42:11.184 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T02:42:11.184 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T02:42:11.184 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T02:42:11.184 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T02:42:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:11.461 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T02:42:11.467 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:11.468 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:11.476 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:42:11.477 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:42:11.477 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T02:42:11.477 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:11.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse 2024-09-15T02:42:12.303 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:12.425 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:42:12.425 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:12.425 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:12.426 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:42:12.426 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:12.426 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:12.442 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:12.442 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:12.442 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:42:12.442 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:12.442 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:12.442 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T02:42:12.442 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-15T02:42:12.443 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T02:42:12.443 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:12.443 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:12.443 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:12.443 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-09-15T02:42:12.443 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:12.443 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 893 k 2024-09-15T02:42:12.444 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 2.6 M 2024-09-15T02:42:12.444 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:12.722 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2024-09-15T02:42:12.731 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 497 kB/s | 80 kB 00:00 2024-09-15T02:42:12.865 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-15T02:42:12.865 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:12.865 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.1 MB/s | 893 kB 00:00 2024-09-15T02:42:12.878 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:12.902 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:12.902 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:12.985 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:12.986 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:13.183 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:13.311 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:13.363 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T02:42:13.433 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:42:13.443 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:13.443 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:13.443 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-15T02:42:13.443 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:13.443 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:13.443 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T02:42:13.444 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:13.444 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:13.444 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:13.444 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:42:13.444 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:13.444 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 126 k 2024-09-15T02:42:13.445 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 448 k 2024-09-15T02:42:13.445 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 601 kB/s | 126 kB 00:00 2024-09-15T02:42:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:Total 597 kB/s | 126 kB 00:00 2024-09-15T02:42:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:13.724 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:13.724 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:13.872 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:13.959 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:14.120 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:42:14.120 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T02:42:14.392 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T02:42:14.392 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:14.393 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:14.393 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T02:42:14.393 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:14.393 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:14.400 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:14.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-volume 2024-09-15T02:42:14.669 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:14.669 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:14.669 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:14.670 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:14.670 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:14.670 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:14.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2024-09-15T02:42:15.257 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:15.396 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:15.396 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:15.396 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 270 k 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.3 M 2024-09-15T02:42:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:15.490 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:15.581 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:15.624 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:15.625 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:42:15.625 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:15.643 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-15T02:42:15.644 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:15.644 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-15T02:42:15.644 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:15.653 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:15.653 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:15.687 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:15.687 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:15.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2024-09-15T02:42:15.908 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:15.970 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:42:15.987 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:42:15.987 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:42:15.987 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T02:42:15.987 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:16.255 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:16.387 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:16.388 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-15T02:42:16.388 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:16.388 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:42:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:16.389 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:16.389 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:16.389 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:42:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:16.389 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 25 k 2024-09-15T02:42:16.389 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 110 k 2024-09-15T02:42:16.389 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:16.549 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 156 kB/s | 25 kB 00:00 2024-09-15T02:42:16.550 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:16.550 INFO:teuthology.orchestra.run.smithi104.stdout:Total 154 kB/s | 25 kB 00:00 2024-09-15T02:42:16.550 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:16.553 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:16.554 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:16.555 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:16.555 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:16.635 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:16.755 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:17.099 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:42:17.099 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:17.099 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:17.099 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:42:17.099 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:17.099 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:17.165 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:17.398 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:17.398 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:17.398 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:17.398 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:17.398 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:17.398 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:17.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel 2024-09-15T02:42:17.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2024-09-15T02:42:18.016 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:18.154 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:18.155 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:18.155 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 126 k 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 448 k 2024-09-15T02:42:18.156 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:18.163 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:18.252 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:18.295 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:18.296 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:42:18.296 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:18.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2024-09-15T02:42:18.373 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 583 kB/s | 126 kB 00:00 2024-09-15T02:42:18.373 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:18.374 INFO:teuthology.orchestra.run.smithi053.stdout:Total 579 kB/s | 126 kB 00:00 2024-09-15T02:42:18.374 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:18.380 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:18.380 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:18.446 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:18.447 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:18.607 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:18.715 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:18.934 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:19.030 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:19.072 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:19.073 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:42:19.073 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:19.139 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2024-09-15T02:42:19.165 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:19.441 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:19.441 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:19.442 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:19.442 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:19.442 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:19.442 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:19.701 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:19.791 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:19.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2 2024-09-15T02:42:19.832 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:19.833 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:42:19.834 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:19.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2024-09-15T02:42:20.409 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:20.470 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:20.501 INFO:teuthology.orchestra.run.smithi053.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:20.546 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:20.547 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:42:20.548 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:20.559 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:20.601 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:20.602 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:42:20.603 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:20.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs-devel 2024-09-15T02:42:20.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2024-09-15T02:42:21.213 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:21.218 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:21.307 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:21.349 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:21.350 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:42:21.350 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:21.352 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:42:21.353 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:21.354 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 25 k 2024-09-15T02:42:21.354 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 110 k 2024-09-15T02:42:21.354 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:21.411 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2024-09-15T02:42:21.502 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 169 kB/s | 25 kB 00:00 2024-09-15T02:42:21.502 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:21.502 INFO:teuthology.orchestra.run.smithi053.stdout:Total 168 kB/s | 25 kB 00:00 2024-09-15T02:42:21.502 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:21.598 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:21.701 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:21.966 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:22.055 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:22.097 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:22.098 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-15T02:42:22.098 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:22.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2024-09-15T02:42:22.179 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:22.415 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:22.415 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:22.415 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:22.415 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:22.415 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:22.415 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:22.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2 2024-09-15T02:42:22.727 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:22.859 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:22.859 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 82 k 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 222 k 2024-09-15T02:42:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:23.035 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 468 kB/s | 82 kB 00:00 2024-09-15T02:42:23.036 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:23.036 INFO:teuthology.orchestra.run.smithi104.stdout:Total 464 kB/s | 82 kB 00:00 2024-09-15T02:42:23.036 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:23.043 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:23.043 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:23.104 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:23.104 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:23.189 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:23.282 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:23.328 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:23.329 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:42:23.329 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:23.365 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:23.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1 2024-09-15T02:42:23.763 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:23.996 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:24.011 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:24.011 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:24.012 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:24.012 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:24.012 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:24.012 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:24.136 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:24.137 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:42:24.137 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:24.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados 2024-09-15T02:42:24.269 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2024-09-15T02:42:24.812 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:24.846 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:24.907 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:24.952 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:24.953 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:42:24.953 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:24.981 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:24.982 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:24.982 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-15T02:42:24.982 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:24.982 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.9 M 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 12 M 2024-09-15T02:42:24.983 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:25.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw 2024-09-15T02:42:25.313 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.9 MB/s | 2.9 MB 00:00 2024-09-15T02:42:25.314 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:25.314 INFO:teuthology.orchestra.run.smithi104.stdout:Total 8.9 MB/s | 2.9 MB 00:00 2024-09-15T02:42:25.315 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:25.323 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:25.323 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:25.403 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:25.403 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:25.619 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:25.711 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:25.756 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:25.757 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:42:25.757 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:25.816 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:25.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs 2024-09-15T02:42:25.872 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:25.896 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:25.896 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:42:25.897 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T02:42:25.897 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:42:25.897 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:42:25.897 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:26.427 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:26.519 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:26.565 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:26.566 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:42:26.566 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:26.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd 2024-09-15T02:42:26.963 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:26.964 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:26.964 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:26.964 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:26.964 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:26.964 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:27.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2024-09-15T02:42:27.229 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:27.323 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:42:27.368 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:27.369 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-15T02:42:27.369 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:27.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse 2024-09-15T02:42:27.812 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 02:40:47 AM UTC. 2024-09-15T02:42:27.947 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-15T02:42:27.947 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:27.947 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-15T02:42:27.947 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:27.948 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-15T02:42:27.948 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T02:42:27.948 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:27.948 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-15T02:42:27.948 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-15T02:42:27.948 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-15T02:42:27.948 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:27.949 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 165 k 2024-09-15T02:42:27.949 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 477 k 2024-09-15T02:42:27.949 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-15T02:42:28.048 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:28.166 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 760 kB/s | 165 kB 00:00 2024-09-15T02:42:28.167 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:28.167 INFO:teuthology.orchestra.run.smithi104.stdout:Total 754 kB/s | 165 kB 00:00 2024-09-15T02:42:28.167 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-15T02:42:28.175 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-15T02:42:28.175 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-15T02:42:28.189 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:28.190 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:28.190 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-09-15T02:42:28.190 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:28.190 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:28.190 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T02:42:28.190 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:28.191 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:28.191 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:28.191 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:42:28.191 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:28.191 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 82 k 2024-09-15T02:42:28.191 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 222 k 2024-09-15T02:42:28.191 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:28.238 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-15T02:42:28.239 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-15T02:42:28.389 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 415 kB/s | 82 kB 00:00 2024-09-15T02:42:28.389 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:28.390 INFO:teuthology.orchestra.run.smithi053.stdout:Total 410 kB/s | 82 kB 00:00 2024-09-15T02:42:28.390 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:28.392 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-15T02:42:28.397 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:28.398 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:28.464 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:28.465 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:28.496 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:28.617 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:28.739 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:28.905 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:29.188 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:29.188 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-15T02:42:29.188 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:29.188 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:29.188 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-15T02:42:29.199 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:29.446 DEBUG:teuthology.parallel:result is None 2024-09-15T02:42:29.497 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:29.497 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:29.497 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:29.497 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:29.497 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:29.498 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:29.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2024-09-15T02:42:30.484 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:30.624 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:30.624 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.9 M 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 12 M 2024-09-15T02:42:30.625 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:30.975 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.4 MB/s | 2.9 MB 00:00 2024-09-15T02:42:30.976 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:30.976 INFO:teuthology.orchestra.run.smithi053.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-09-15T02:42:30.976 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:30.986 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:30.986 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:31.070 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:31.070 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:31.510 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:31.584 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:31.612 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:31.612 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:42:31.612 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T02:42:31.612 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:42:31.612 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:42:31.612 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:32.756 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:32.756 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:32.756 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:32.756 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:32.756 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:32.756 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:33.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd 2024-09-15T02:42:33.628 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 02:40:46 AM UTC. 2024-09-15T02:42:33.769 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-15T02:42:33.769 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:33.769 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 165 k 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 477 k 2024-09-15T02:42:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-15T02:42:33.978 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 796 kB/s | 165 kB 00:00 2024-09-15T02:42:33.978 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:42:33.979 INFO:teuthology.orchestra.run.smithi053.stdout:Total 790 kB/s | 165 kB 00:00 2024-09-15T02:42:33.979 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-15T02:42:33.987 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-15T02:42:33.988 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-15T02:42:34.054 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-15T02:42:34.055 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-15T02:42:34.217 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-15T02:42:34.312 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:34.735 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:35.006 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:42:35.006 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:35.006 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-15T02:42:35.006 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:42:35.006 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:35.006 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-15T02:42:35.267 DEBUG:teuthology.parallel:result is None 2024-09-15T02:42:35.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:42:35.418 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T02:42:35.440 INFO:teuthology.orchestra.run.smithi053.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T02:42:35.441 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T02:42:35.441 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T02:42:35.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:42:35.587 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T02:42:35.614 INFO:teuthology.orchestra.run.smithi104.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T02:42:35.614 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T02:42:35.614 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T02:42:35.616 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T02:42:35.616 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:35.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:42:35.643 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:42:35.643 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:42:35.684 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T02:42:35.685 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:35.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:42:35.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:42:35.778 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:42:35.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:42:35.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:42:35.872 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T02:42:35.873 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:35.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:42:35.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:42:35.964 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:42:35.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:42:35.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:42:36.055 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T02:42:36.055 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:36.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:42:36.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:42:36.146 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:42:36.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:42:36.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:42:36.236 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T02:42:36.312 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T02:42:36.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2024-09-15T02:42:36.315 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2024-09-15T02:42:36.339 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T02:42:36.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T02:42:36.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T02:42:36.406 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T02:42:36.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T02:42:36.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T02:42:36.485 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T02:42:36.485 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T02:42:36.485 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905170', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBFdla0bUvc8hN2PIJ0soM2p00rcrdYe3ECnNfHVq8kVgROECIA+yJMp4NlwQ8u6EYK1AbbjMx7RlNHLi4/smk=', 'smithi104.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBzgR2t8fqyln5yFzb62a4CsxCsWAtIlw4T6Sfxjzjif4x3ZZbLl+oUtA005ZN63cWRKKiQxm/GFsVpJZXwh4+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T02:42:36.486 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T02:42:36.517 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T02:42:36.533 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T02:42:36.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T02:42:36.598 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:36.599 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T02:42:36.655 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T02:42:36.655 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-09-15T02:42:36.711 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T02:42:36.711 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:36.711 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-15T02:42:36.711 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:36.711 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:36.712 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-15 02:42:31.870529443 +0000 2024-09-15T02:42:36.712 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-15 02:40:06.856990233 +0000 2024-09-15T02:42:36.712 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-15 02:40:06.856990233 +0000 2024-09-15T02:42:36.712 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-15 02:40:06.856990233 +0000 2024-09-15T02:42:36.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T02:42:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-15T02:42:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-15T02:42:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000102132 s, 5.0 MB/s 2024-09-15T02:42:36.776 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T02:42:36.833 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-09-15T02:42:36.889 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-15 02:42:31.871529424 +0000 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-15 02:40:07.183983932 +0000 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-15 02:40:07.183983932 +0000 2024-09-15T02:42:36.890 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-15 02:40:07.183983932 +0000 2024-09-15T02:42:36.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T02:42:36.954 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-15T02:42:36.954 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-15T02:42:36.954 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 8.4348e-05 s, 6.1 MB/s 2024-09-15T02:42:36.955 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T02:42:37.013 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-15 02:42:31.871529424 +0000 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-15 02:40:07.515977534 +0000 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-15 02:40:07.515977534 +0000 2024-09-15T02:42:37.069 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-15 02:40:07.515977534 +0000 2024-09-15T02:42:37.070 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T02:42:37.132 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-15T02:42:37.133 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-15T02:42:37.133 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.7048e-05 s, 5.3 MB/s 2024-09-15T02:42:37.134 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T02:42:37.191 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-15 02:42:31.871529424 +0000 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-15 02:40:07.852971040 +0000 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-15 02:40:07.852971040 +0000 2024-09-15T02:42:37.247 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-15 02:40:07.852971040 +0000 2024-09-15T02:42:37.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T02:42:37.310 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-15T02:42:37.311 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-15T02:42:37.311 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000101649 s, 5.0 MB/s 2024-09-15T02:42:37.313 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T02:42:37.370 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:37.371 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:42:37.371 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T02:42:37.387 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T02:42:37.387 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-09-15T02:42:37.442 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-15 02:42:26.148204893 +0000 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-15 02:40:02.834188834 +0000 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-15 02:40:02.834188834 +0000 2024-09-15T02:42:37.443 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-15 02:40:02.834188834 +0000 2024-09-15T02:42:37.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T02:42:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-15T02:42:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-15T02:42:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000208576 s, 2.5 MB/s 2024-09-15T02:42:37.508 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T02:42:37.564 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-15 02:42:26.148204893 +0000 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-15 02:40:03.161184995 +0000 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-15 02:40:03.161184995 +0000 2024-09-15T02:42:37.619 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-15 02:40:03.161184995 +0000 2024-09-15T02:42:37.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T02:42:37.685 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-15T02:42:37.685 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-15T02:42:37.685 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000179375 s, 2.9 MB/s 2024-09-15T02:42:37.685 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T02:42:37.741 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-09-15T02:42:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T02:42:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:37.796 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T02:42:37.797 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:37.797 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:37.797 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-15 02:42:26.148204893 +0000 2024-09-15T02:42:37.797 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-15 02:40:03.494181086 +0000 2024-09-15T02:42:37.797 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-15 02:40:03.494181086 +0000 2024-09-15T02:42:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-15 02:40:03.494181086 +0000 2024-09-15T02:42:37.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T02:42:37.859 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-15T02:42:37.859 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-15T02:42:37.859 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000175438 s, 2.9 MB/s 2024-09-15T02:42:37.860 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T02:42:37.916 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-09-15T02:42:37.971 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T02:42:37.971 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:42:37.971 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-15T02:42:37.971 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:42:37.971 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T02:42:37.971 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-15 02:42:26.149204880 +0000 2024-09-15T02:42:37.971 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-15 02:40:03.819177270 +0000 2024-09-15T02:42:37.972 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-15 02:40:03.819177270 +0000 2024-09-15T02:42:37.972 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-15 02:40:03.819177270 +0000 2024-09-15T02:42:37.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T02:42:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-15T02:42:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-15T02:42:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000214076 s, 2.4 MB/s 2024-09-15T02:42:38.035 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T02:42:38.091 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:38.091 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-15T02:42:38.091 INFO:tasks.ceph:Generating config... 2024-09-15T02:42:38.093 INFO:tasks.ceph:[client] debug monc = 20 2024-09-15T02:42:38.093 INFO:tasks.ceph:[client] debug ms = 1 2024-09-15T02:42:38.093 INFO:tasks.ceph:[client] debug objecter = 20 2024-09-15T02:42:38.093 INFO:tasks.ceph:[client] debug rados = 20 2024-09-15T02:42:38.093 INFO:tasks.ceph:[global] lockdep = True 2024-09-15T02:42:38.093 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T02:42:38.093 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T02:42:38.093 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-15T02:42:38.093 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-15T02:42:38.093 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-15T02:42:38.093 INFO:tasks.ceph:[global] ms type = async 2024-09-15T02:42:38.093 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T02:42:38.093 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T02:42:38.094 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T02:42:38.094 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T02:42:38.094 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T02:42:38.094 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-15T02:42:38.096 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-15T02:42:38.096 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-15T02:42:38.096 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-15T02:42:38.096 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T02:42:38.096 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-15T02:42:38.096 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T02:42:38.097 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T02:42:38.098 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T02:42:38.099 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-15T02:42:38.099 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T02:42:38.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T02:42:38.153 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T02:42:38.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T02:42:38.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:42:38.266 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.53'), ('mon.c', '[v2:172.21.15.53:3301,v1:172.21.15.53:6790]'), ('mon.b', '172.21.15.104')] 2024-09-15T02:42:38.267 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790],172.21.15.104', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-15T02:42:38.268 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:38.268 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T02:42:38.323 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.53 --addv c '[v2:172.21.15.53:3301,v1:172.21.15.53:6790]' --add b 172.21.15.104 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:38.407 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:38.407 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: generated fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:setting min_mon_release = octopus 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 0 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:min_mon_release 15 (octopus) 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:election_strategy: 1 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:42:38.408 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T02:42:38.410 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T02:42:38.466 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7e7cb607-2b58-4a36-a082-99425792b70f... 2024-09-15T02:42:38.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T02:42:38.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T02:42:38.550 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-09-15T02:42:38.550 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-09-15T02:42:38.550 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-09-15T02:42:38.551 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-09-15T02:42:38.552 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.553 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790],172.21.15.104" 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: lockdep = True 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: mon client directed command retry = 5 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: mon election default strategy = 1 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: ms bind msgr1 = False 2024-09-15T02:42:38.554 INFO:teuthology.orchestra.run.smithi053.stdout: ms bind msgr2 = True 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: ms inject socket failures = 5000 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: ms type = async 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-09-15T02:42:38.555 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore compression algorithm = zlib 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore compression mode = aggressive 2024-09-15T02:42:38.557 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore zero block detection = True 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 20 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 20 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: debug monc = 20 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 10 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-09-15T02:42:38.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-09-15T02:42:38.559 INFO:teuthology.orchestra.run.smithi053.stdout: osd class default list = * 2024-09-15T02:42:38.559 INFO:teuthology.orchestra.run.smithi053.stdout: osd class load list = * 2024-09-15T02:42:38.559 INFO:teuthology.orchestra.run.smithi053.stdout: osd client watch timeout = 120 2024-09-15T02:42:38.559 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify cached snaps = True 2024-09-15T02:42:38.559 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify missing on start = True 2024-09-15T02:42:38.559 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-09-15T02:42:38.559 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock override recovery settings = True 2024-09-15T02:42:38.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-09-15T02:42:38.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd op queue = debug_random 2024-09-15T02:42:38.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd op queue cut off = debug_random 2024-09-15T02:42:38.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd shutdown pgref assert = True 2024-09-15T02:42:38.560 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.560 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-09-15T02:42:38.561 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-15T02:42:38.561 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-09-15T02:42:38.561 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-09-15T02:42:38.561 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-09-15T02:42:38.561 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:42:38.561 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:42:38.562 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-09-15T02:42:38.562 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.562 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-09-15T02:42:38.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-15T02:42:38.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-09-15T02:42:38.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-09-15T02:42:38.563 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-09-15T02:42:38.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-09-15T02:42:38.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-09-15T02:42:38.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:42:38.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-09-15T02:42:38.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-09-15T02:42:38.564 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:42:38.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.564 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T02:42:38.564 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T02:42:38.564 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T02:42:38.565 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.565 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T02:42:38.565 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T02:42:38.565 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T02:42:38.565 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.565 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-09-15T02:42:38.566 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-09-15T02:42:38.566 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.566 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-09-15T02:42:38.566 INFO:teuthology.orchestra.run.smithi053.stdout: mon min osdmap epochs = 50 2024-09-15T02:42:38.566 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune interval = 2 2024-09-15T02:42:38.566 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune min = 15 2024-09-15T02:42:38.567 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune txsize = 2 2024-09-15T02:42:38.567 INFO:teuthology.orchestra.run.smithi053.stdout: mon scrub interval = 300 2024-09-15T02:42:38.567 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on pool no app = False 2024-09-15T02:42:38.567 INFO:teuthology.orchestra.run.smithi053.stdout: paxos service trim min = 10 2024-09-15T02:42:38.567 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:42:38.567 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-09-15T02:42:38.568 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-09-15T02:42:38.568 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-09-15T02:42:38.568 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-09-15T02:42:38.568 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T02:42:38.568 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T02:42:38.568 INFO:teuthology.orchestra.run.smithi053.stdout: debug monc = 20 2024-09-15T02:42:38.569 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-15T02:42:38.569 INFO:teuthology.orchestra.run.smithi053.stdout: debug objecter = 20 2024-09-15T02:42:38.569 INFO:teuthology.orchestra.run.smithi053.stdout: debug rados = 20 2024-09-15T02:42:38.569 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-09-15T02:42:38.569 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-09-15T02:42:38.569 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-09-15T02:42:38.572 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-09-15T02:42:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2024-09-15T02:42:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T02:42:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2024-09-15T02:42:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2024-09-15T02:42:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2024-09-15T02:42:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2024-09-15T02:42:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2024-09-15T02:42:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2024-09-15T02:42:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2024-09-15T02:42:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T02:42:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2024-09-15T02:42:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2024-09-15T02:42:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: mon_allow_pool_size_one = true 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2024-09-15T02:42:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: mon health detail to clog = false 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790],172.21.15.104" 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: lockdep = True 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: mon client directed command retry = 5 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: mon election default strategy = 1 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: ms bind msgr1 = False 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: ms bind msgr2 = True 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject socket failures = 5000 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: ms type = async 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2024-09-15T02:42:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2024-09-15T02:42:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2024-09-15T02:42:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 600 2024-09-15T02:42:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock profile = high_recovery_ops 2024-09-15T02:42:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2024-09-15T02:42:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2024-09-15T02:42:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore block size = 96636764160 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression algorithm = zlib 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression mode = aggressive 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore fsck on mount = True 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore zero block detection = True 2024-09-15T02:42:38.582 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluefs = 20 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluestore = 20 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 20 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: debug rocksdb = 10 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd full ratio = 0.9 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd nearfull ratio = 0.8 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: osd class default list = * 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: osd class load list = * 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: osd client watch timeout = 120 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify cached snaps = True 2024-09-15T02:42:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify missing on start = True 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: osd failsafe full ratio = 0.95 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock override recovery settings = True 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = bluestore 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue = debug_random 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue cut off = debug_random 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: osd shutdown pgref assert = True 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:42:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2024-09-15T02:42:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: # 1m isn't quite enough 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: mon_down_mkfs_grace = 2m 2024-09-15T02:42:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_filestore_osds = false 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: mon min osdmap epochs = 50 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune interval = 2 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune min = 15 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune txsize = 2 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub interval = 300 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on pool no app = False 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: paxos service trim min = 10 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2024-09-15T02:42:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: debug objecter = 20 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: debug rados = 20 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2024-09-15T02:42:38.588 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2024-09-15T02:42:38.589 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T02:42:38.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T02:42:38.663 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T02:42:38.663 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:38.663 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T02:42:38.679 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:38.680 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T02:42:38.735 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:42:38.736 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:38.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T02:42:38.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:42:38.811 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:42:38.812 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:38.868 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:42:38.868 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:42:38.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T02:42:38.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:42:38.903 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:42:38.903 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:38.960 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T02:42:38.960 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T02:42:38.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-15T02:42:39.009 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-15T02:42:39.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T02:42:39.065 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T02:42:39.067 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T02:42:39.068 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T02:42:39.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T02:42:39.106 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T02:42:39.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-15T02:42:39.157 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-15T02:42:39.168 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T02:42:39.169 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-15T02:42:39.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T02:42:39.194 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:39.194 INFO:tasks.ceph:role: osd.0 2024-09-15T02:42:39.194 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:42:39.195 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T02:42:39.261 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:39.261 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:39.261 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:39.261 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:39.261 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:39.261 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:39.262 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:39.262 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:39.262 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:39.262 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:39.293 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-09-15T02:42:39.295 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-09-15T02:42:39.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T02:42:39.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T02:42:39.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T02:42:39.503 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:39.503 INFO:tasks.ceph:role: osd.1 2024-09-15T02:42:39.503 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:42:39.503 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T02:42:39.568 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:39.568 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:39.568 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:39.568 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:39.568 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:39.568 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:39.569 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:39.569 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:39.569 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:39.569 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:39.600 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-09-15T02:42:39.602 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-09-15T02:42:39.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T02:42:39.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T02:42:39.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T02:42:39.809 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:39.809 INFO:tasks.ceph:role: osd.2 2024-09-15T02:42:39.809 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:42:39.809 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T02:42:39.875 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:39.875 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:39.875 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:39.875 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:39.875 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:39.876 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:39.876 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:39.876 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:39.876 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:39.876 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:39.908 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-09-15T02:42:39.910 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-09-15T02:42:39.911 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T02:42:39.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T02:42:40.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T02:42:40.117 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:40.117 INFO:tasks.ceph:role: osd.3 2024-09-15T02:42:40.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:42:40.117 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T02:42:40.182 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:40.182 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:40.183 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:40.215 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-09-15T02:42:40.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-09-15T02:42:40.217 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-15T02:42:40.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T02:42:40.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:40.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:40.474+0000 7f68b566f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T02:42:40.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:40.474+0000 7f68b566f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T02:42:40.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:40.474+0000 7f68b566f440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T02:42:40.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:40.474+0000 7f68b566f440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T02:42:40.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:40.474+0000 7f68b566f440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T02:42:40.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:40.475+0000 7f68b566f440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T02:42:40.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:40.996+0000 7f68b566f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:41.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:41.765+0000 7f68b566f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:42.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:42.533+0000 7f68b566f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:43.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T02:42:43.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:43.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:43.445+0000 7f501fb55440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T02:42:43.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:43.445+0000 7f501fb55440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T02:42:43.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:43.445+0000 7f501fb55440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T02:42:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:43.445+0000 7f501fb55440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T02:42:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:43.445+0000 7f501fb55440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T02:42:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:43.445+0000 7f501fb55440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T02:42:43.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:43.969+0000 7f501fb55440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:44.734+0000 7f501fb55440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:45.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:45.504+0000 7f501fb55440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:46.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T02:42:46.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:46.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:46.426+0000 7fe4d220f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T02:42:46.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:46.426+0000 7fe4d220f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T02:42:46.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:46.426+0000 7fe4d220f440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T02:42:46.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:46.426+0000 7fe4d220f440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T02:42:46.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:46.426+0000 7fe4d220f440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T02:42:46.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:46.426+0000 7fe4d220f440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T02:42:46.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:46.951+0000 7fe4d220f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:47.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:47.712+0000 7fe4d220f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:48.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:48.475+0000 7fe4d220f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:49.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T02:42:49.301 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:49.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:49.388+0000 7fd937639440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T02:42:49.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:49.388+0000 7fd937639440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T02:42:49.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:49.388+0000 7fd937639440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T02:42:49.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:49.388+0000 7fd937639440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T02:42:49.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:49.388+0000 7fd937639440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T02:42:49.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:49.388+0000 7fd937639440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T02:42:49.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:49.912+0000 7fd937639440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:50.672+0000 7fd937639440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:51.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:42:51.435+0000 7fd937639440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:52.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T02:42:52.262 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T02:42:52.292 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:52.293 INFO:tasks.ceph:role: osd.4 2024-09-15T02:42:52.293 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:42:52.293 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:52.370 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:52.371 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:52.870 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-09-15T02:42:52.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-09-15T02:42:52.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-15T02:42:53.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T02:42:53.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T02:42:53.126 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:53.126 INFO:tasks.ceph:role: osd.5 2024-09-15T02:42:53.126 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:42:53.126 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T02:42:53.199 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:53.200 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:53.643 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-09-15T02:42:53.646 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-09-15T02:42:53.646 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-15T02:42:53.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T02:42:53.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-15T02:42:53.835 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:53.835 INFO:tasks.ceph:role: osd.6 2024-09-15T02:42:53.835 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:42:53.835 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T02:42:53.907 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:53.908 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:54.387 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-09-15T02:42:54.389 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-09-15T02:42:54.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-15T02:42:54.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-15T02:42:54.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-15T02:42:54.620 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T02:42:54.620 INFO:tasks.ceph:role: osd.7 2024-09-15T02:42:54.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:42:54.620 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T02:42:54.692 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:42:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:42:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:42:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T02:42:54.692 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:42:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:42:54.692 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:42:54.693 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T02:42:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:42:54.693 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:42:55.213 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-09-15T02:42:55.215 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-09-15T02:42:55.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-15T02:42:55.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-15T02:42:55.401 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:55.497+0000 7f84ea23f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T02:42:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:55.497+0000 7f84ea23f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T02:42:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:55.497+0000 7f84ea23f440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T02:42:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:55.497+0000 7f84ea23f440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T02:42:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:55.497+0000 7f84ea23f440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T02:42:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:55.497+0000 7f84ea23f440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-15T02:42:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:56.022+0000 7f84ea23f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:56.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:56.790+0000 7f84ea23f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:57.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:57.553+0000 7f84ea23f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:58.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T02:42:58.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:42:58.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:58.465+0000 7f9d9054c440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T02:42:58.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:58.465+0000 7f9d9054c440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T02:42:58.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:58.465+0000 7f9d9054c440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T02:42:58.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:58.465+0000 7f9d9054c440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T02:42:58.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:58.465+0000 7f9d9054c440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T02:42:58.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:58.466+0000 7f9d9054c440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-15T02:42:58.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:58.991+0000 7f9d9054c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:42:59.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:42:59.753+0000 7f9d9054c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:00.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:00.515+0000 7f9d9054c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:01.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T02:43:01.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:43:01.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:01.425+0000 7f7d9751f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-15T02:43:01.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:01.426+0000 7f7d9751f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-15T02:43:01.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:01.426+0000 7f7d9751f440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T02:43:01.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:01.426+0000 7f7d9751f440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T02:43:01.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:01.426+0000 7f7d9751f440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T02:43:01.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:01.426+0000 7f7d9751f440 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-15T02:43:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:01.953+0000 7f7d9751f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:02.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:02.718+0000 7f7d9751f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:03.482+0000 7f7d9751f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:04.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-15T02:43:04.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:43:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:04.394+0000 7ff2da20d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-15T02:43:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:04.394+0000 7ff2da20d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-15T02:43:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:04.394+0000 7ff2da20d440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T02:43:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:04.395+0000 7ff2da20d440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T02:43:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:04.395+0000 7ff2da20d440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T02:43:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:04.395+0000 7ff2da20d440 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-15T02:43:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:04.921+0000 7ff2da20d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:05.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:05.685+0000 7ff2da20d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:06.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:06.447+0000 7ff2da20d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:43:07.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-15T02:43:07.273 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T02:43:07.273 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:07.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-15T02:43:07.306 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:07.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T02:43:07.377 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:07.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T02:43:07.449 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:07.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T02:43:07.517 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:07.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T02:43:07.589 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:07.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T02:43:07.619 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:07.619 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T02:43:07.688 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:07.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T02:43:07.756 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:07.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-15T02:43:07.822 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:07.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-15T02:43:07.889 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:07.889 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T02:43:07.905 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:07.905 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-15T02:43:07.945 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T02:43:07.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T02:43:07.947 DEBUG:teuthology.orchestra.run.smithi104:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T02:43:07.988 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.y] 2024-09-15T02:43:07.988 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCfSeZmMGSQABAABWmqb5BG1ga54mp29JJN6Q== 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCgSeZmNNxUHBAAc/Mz+AZ0F5iZNP1S70POfQ== 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCjSeZmNX6UGhAAQ+Qm6AX22Xa98G2yKBYiIg== 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCmSeZmt95wGRAAfO9faC0sgMT6XfQr5sOdIw== 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.3] 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCpSeZmTpgsFxAAMxxfm0+shN2hHBOCmARvsQ== 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCfSeZmYsHiAxAAG2FDS7ZvigEsxMJFQxgvhA== 2024-09-15T02:43:07.989 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.4] 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCvSeZmfxGuHRAAaQpSGl/cH6ABqL5J17V2XA== 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.5] 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCySeZmR2bOGxAAyZ5IQU2Q2EiMPWky6PzGnw== 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.6] 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC1SeZmPAhyGRAAUTiYJ0EI4NTZ7oYt3Uf+bw== 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.7] 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC4SeZm/tCTFxAAzr73MZ5gg7N095a3v8MrJw== 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCfSeZmI9dVBhAAHo/VChE+zDHH7IFjtQ6Elg== 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout:[client.1] 2024-09-15T02:43:07.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCfSeZmdYtWCRAAgI2DOTjmyYoUkqRqEbjxZQ== 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.y] 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCfSeZmMGSQABAABWmqb5BG1ga54mp29JJN6Q== 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.0] 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCgSeZmNNxUHBAAc/Mz+AZ0F5iZNP1S70POfQ== 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.1] 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCjSeZmNX6UGhAAQ+Qm6AX22Xa98G2yKBYiIg== 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.2] 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCmSeZmt95wGRAAfO9faC0sgMT6XfQr5sOdIw== 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.3] 2024-09-15T02:43:08.009 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCpSeZmTpgsFxAAMxxfm0+shN2hHBOCmARvsQ== 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.x] 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCfSeZmYsHiAxAAG2FDS7ZvigEsxMJFQxgvhA== 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.4] 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCvSeZmfxGuHRAAaQpSGl/cH6ABqL5J17V2XA== 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.5] 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCySeZmR2bOGxAAyZ5IQU2Q2EiMPWky6PzGnw== 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.6] 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQC1SeZmPAhyGRAAUTiYJ0EI4NTZ7oYt3Uf+bw== 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.7] 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQC4SeZm/tCTFxAAzr73MZ5gg7N095a3v8MrJw== 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCfSeZmI9dVBhAAHo/VChE+zDHH7IFjtQ6Elg== 2024-09-15T02:43:08.010 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2024-09-15T02:43:08.011 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCfSeZmdYtWCRAAgI2DOTjmyYoUkqRqEbjxZQ== 2024-09-15T02:43:08.011 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:43:08.034 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:43:08.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.152 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:43:08.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:43:08.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.490 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:43:08.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T02:43:08.538 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T02:43:08.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T02:43:08.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T02:43:08.638 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T02:43:08.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T02:43:08.664 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T02:43:08.756 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T02:43:08.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T02:43:08.848 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T02:43:08.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T02:43:08.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T02:43:08.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T02:43:09.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T02:43:09.106 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:43:09.108 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:43:09.161 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T02:43:09.162 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T02:43:09.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T02:43:09.164 INFO:tasks.ceph.mon.a:Started 2024-09-15T02:43:09.164 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T02:43:09.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T02:43:09.166 INFO:tasks.ceph.mon.c:Started 2024-09-15T02:43:09.167 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:43:09.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:43:09.204 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:43:09.205 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T02:43:09.205 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-15T02:43:09.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-15T02:43:09.207 INFO:tasks.ceph.mgr.y:Started 2024-09-15T02:43:09.208 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T02:43:09.208 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T02:43:09.210 INFO:tasks.ceph.mgr.x:Started 2024-09-15T02:43:09.210 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T02:43:09.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T02:43:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.306+0000 7f009caef640 10 monclient: get_monmap_and_config 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.306+0000 7f009caef640 10 monclient: build_initial_monmap 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.306+0000 7f009caef640 10 monclient: monmap: 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:09.308176+0000 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:09.308176+0000 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:09.310 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 1 Processor -- start 2024-09-15T02:43:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 1 -- start start 2024-09-15T02:43:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient: init 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 0x7f00980b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient(hunting): picked mon.noname-a con 0x7f00980b0e80 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 0x7f009813def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient(hunting): picked mon.noname-c con 0x7f00980b1250 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f009813e430 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009813da60 con 0x7f009813e430 2024-09-15T02:43:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00980b0990 con 0x7f00980b0e80 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009813e890 con 0x7f00980b1250 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009caef640 10 monclient(hunting): authenticate will time out at 725.878871s 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.307+0000 7f009659c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 0x7f00980b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 0x7f00980b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39642/0 (socket says 172.21.15.53:39642) 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 1 -- 172.21.15.53:0/778810146 learned_addr learned my addr 172.21.15.53:0/778810146 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 10 monclient(hunting): get_auth_request con 0x7f00980b0e80 auth_method 0 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 1 -- 172.21.15.53:0/778810146 >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 msgr2=0x7f00980b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 1 -- 172.21.15.53:0/778810146 >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 msgr2=0x7f00980b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 1 --2- 172.21.15.53:0/778810146 >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 0x7f00980b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 1 --2- 172.21.15.53:0/778810146 >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 0x7f009813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f009659c640 1 --2- 172.21.15.53:0/778810146 >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 0x7f00980b0450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 10 monclient(hunting): get_auth_request con 0x7f00980b1250 auth_method 0 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 1 -- 172.21.15.53:0/778810146 >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 msgr2=0x7f009813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 1 -- 172.21.15.53:0/778810146 >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 msgr2=0x7f009813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 1 --2- 172.21.15.53:0/778810146 >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 0x7f009813def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:43:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.308+0000 7f0095d9b640 1 --2- 172.21.15.53:0/778810146 >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 0x7f009813def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:43:09.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.471+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652650411 0 0) 0x7f009813da60 con 0x7f009813e430 2024-09-15T02:43:09.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.471+0000 7f009559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.471+0000 7f009559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:09.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.471+0000 7f009559a640 10 monclient(hunting): my global_id is 4099 2024-09-15T02:43:09.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.471+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00800034e0 con 0x7f009813e430 2024-09-15T02:43:09.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762070660 0 0) 0x7f00800034e0 con 0x7f009813e430 2024-09-15T02:43:09.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 msgr2=0x7f009813def0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:43:09.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 --2- 172.21.15.53:0/778810146 >> v2:172.21.15.104:3300/0 conn(0x7f00980b1250 0x7f009813def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 msgr2=0x7f00980b0450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:43:09.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 --2- 172.21.15.53:0/778810146 >> v2:172.21.15.53:3300/0 conn(0x7f00980b0e80 0x7f00980b0450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: _finish_hunting 0 2024-09-15T02:43:09.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:43:09.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:09.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.104:6789/0 2024-09-15T02:43:09.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00980a6a30 con 0x7f009813e430 2024-09-15T02:43:09.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: _finish_auth 0 2024-09-15T02:43:09.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:09.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 65778851 0 0) 0x7f00900017f0 con 0x7f009813e430 2024-09-15T02:43:09.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009caef640 5 monclient: authenticate success, global_id 4099 2024-09-15T02:43:09.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009caef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:09.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:09.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:43:09.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: dump: 2024-09-15T02:43:09.505 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:09.505 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:09.506 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.506 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.506 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:09.506 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.506 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:09.506 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:09.507 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:09.507 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-15T02:43:09.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:09.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 >> v1:172.21.15.104:6789/0 conn(0x7f009813e430 legacy=0x7f00980af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:09.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:43:09.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0080003aa0 0x7f0080003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient(hunting): picked mon.b con 0x7f0080003aa0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:09.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0080004780 0x7f0080004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient(hunting): picked mon.a con 0x7f0080004780 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:09.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080005250 0x7f008000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient(hunting): picked mon.c con 0x7f0080005250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:09.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980b0990 con 0x7f0080004780 2024-09-15T02:43:09.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009813e890 con 0x7f0080003aa0 2024-09-15T02:43:09.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008000c290 con 0x7f0080005250 2024-09-15T02:43:09.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.477+0000 7f009559a640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:09.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009559a640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:43:09.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009caef640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:09.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0080004780 0x7f0080004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0080003aa0 0x7f0080003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): get_auth_request con 0x7f0080003aa0 auth_method 0 2024-09-15T02:43:09.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:09.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 10 monclient(hunting): get_auth_request con 0x7f0080004780 auth_method 0 2024-09-15T02:43:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080005250 0x7f008000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 10 monclient(hunting): get_auth_request con 0x7f0080005250 auth_method 0 2024-09-15T02:43:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2024-09-15T02:43:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 1 -- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080005250 msgr2=0x7f008000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080005250 0x7f008000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 1 -- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0080004780 msgr2=0x7f0080004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0080004780 0x7f0080004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient: _finish_hunting 0 2024-09-15T02:43:09.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 1 monclient: found mon.b 2024-09-15T02:43:09.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:09.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:09.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 1 -- 172.21.15.53:0/778810146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00800034e0 con 0x7f0080003aa0 2024-09-15T02:43:09.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 10 monclient: _finish_auth 0 2024-09-15T02:43:09.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f009659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:09.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0096d9d640 1 -- 172.21.15.53:0/778810146 reap_dead start 2024-09-15T02:43:09.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.478+0000 7f0095d9b640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0080004780 0x7f0080004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:09.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.479+0000 7f009659c640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0080003aa0 0x7f0080003eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0088006fd0 tx=0x7f0088008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e13d8799984cad6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008802acb0 con 0x7f0080003aa0 2024-09-15T02:43:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 10 monclient: dump: 2024-09-15T02:43:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:09.520 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008802ae50 con 0x7f0080003aa0 2024-09-15T02:43:09.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:09.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 1 -- 172.21.15.53:0/778810146 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088032690 con 0x7f0080003aa0 2024-09-15T02:43:09.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009caef640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:09.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009caef640 4 set_mon_vals no callback set 2024-09-15T02:43:09.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:09.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:09.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.481+0000 7f009559a640 10 monclient: dump: 2024-09-15T02:43:09.522 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:09.522 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:09.522 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.522 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:09.524 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 10 monclient: shutdown 2024-09-15T02:43:09.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:09.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 -- 172.21.15.53:0/778810146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0080003aa0 msgr2=0x7f0080003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:09.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0080003aa0 0x7f0080003eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0088006fd0 tx=0x7f0088008040 comp rx=0 tx=0).stop 2024-09-15T02:43:09.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 -- 172.21.15.53:0/778810146 shutdown_connections 2024-09-15T02:43:09.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 --2- 172.21.15.53:0/778810146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0080003aa0 0x7f0080003eb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 -- 172.21.15.53:0/778810146 >> 172.21.15.53:0/778810146 conn(0x7f0098062d90 msgr2=0x7f0098063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:09.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 -- 172.21.15.53:0/778810146 shutdown_connections 2024-09-15T02:43:09.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 -- 172.21.15.53:0/778810146 wait complete. 2024-09-15T02:43:09.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 10 monclient: build_initial_monmap 2024-09-15T02:43:09.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 10 monclient: monmap: 2024-09-15T02:43:09.526 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:09.526 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:09.526 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:09.526 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:09.526 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:09.527 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.527 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:09.527 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:09.527 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:09.527 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.482+0000 7f009caef640 1 librados: starting msgr at 2024-09-15T02:43:09.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 librados: starting objecter 2024-09-15T02:43:09.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 Processor -- start 2024-09-15T02:43:09.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 -- start start 2024-09-15T02:43:09.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 librados: setting wanted keys 2024-09-15T02:43:09.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 librados: calling monclient init 2024-09-15T02:43:09.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient: init 2024-09-15T02:43:09.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:09.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:09.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f00981506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): picked mon.noname-c con 0x7f0080003aa0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:09.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 0x7f00980a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): picked mon.noname-a con 0x7f0098151130 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f00980aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): picked mon.noname-b con 0x7f00980a7150 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980a9e50 con 0x7f0098151130 2024-09-15T02:43:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980a9aa0 con 0x7f00980a7150 2024-09-15T02:43:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00980aa820 con 0x7f0080003aa0 2024-09-15T02:43:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009caef640 10 monclient(hunting): authenticate will time out at 726.054811s 2024-09-15T02:43:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009659c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f00981506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f0095d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 0x7f00980a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009659c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f00981506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:48654/0 (socket says 172.21.15.53:48654) 2024-09-15T02:43:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009659c640 1 -- 172.21.15.53:0/3786999241 learned_addr learned my addr 172.21.15.53:0/3786999241 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009659c640 10 monclient(hunting): get_auth_request con 0x7f0080003aa0 auth_method 0 2024-09-15T02:43:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f009659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f0095d9b640 10 monclient(hunting): get_auth_request con 0x7f0098151130 auth_method 0 2024-09-15T02:43:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f0095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f0095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f0095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.483+0000 7f0096d9d640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f00980aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 10 monclient(hunting): get_auth_request con 0x7f00980a7150 auth_method 0 2024-09-15T02:43:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:09.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:09.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 10 monclient(hunting): handle_auth_done global_id 4110 payload 824 2024-09-15T02:43:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 msgr2=0x7f00981506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f00981506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 msgr2=0x7f00980aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f00980aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:43:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 monclient: found mon.noname-a 2024-09-15T02:43:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00980aa9e0 con 0x7f0098151130 2024-09-15T02:43:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009659c640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f00981506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0096d9d640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f00980aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 0x7f00980a9560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f009806f9b0 tx=0x7f008c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 client.?.objecter ms_handle_connect 0x7f0098151130 2024-09-15T02:43:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c00e780 con 0x7f0098151130 2024-09-15T02:43:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: dump: 2024-09-15T02:43:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.542 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.542 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:09.542 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.542 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:09.542 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:09.543 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:09.543 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:09.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:09.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c00e8e0 con 0x7f0098151130 2024-09-15T02:43:09.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 5 monclient: authenticate success, global_id 4110 2024-09-15T02:43:09.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 10 monclient: _renew_subs 2024-09-15T02:43:09.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:09.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009813a5a0 con 0x7f0098151130 2024-09-15T02:43:09.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:09.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c00ec60 con 0x7f0098151130 2024-09-15T02:43:09.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:09.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:09.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: dump: 2024-09-15T02:43:09.545 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:09.545 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:09.546 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.546 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.546 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:09.546 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.546 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:09.546 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:09.547 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:09.547 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0094d99640 4 set_mon_vals no callback set 2024-09-15T02:43:09.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:09.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 10 monclient: _renew_subs 2024-09-15T02:43:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009813ab40 con 0x7f0098151130 2024-09-15T02:43:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f009caef640 1 librados: init done 2024-09-15T02:43:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 msgr2=0x7f00980a9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:43:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 msgr2=0x7f00980a9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:43:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 0x7f00980a9560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f009806f9b0 tx=0x7f008c008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:43:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f0095d9b640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 0x7f00980a9560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f009806f9b0 tx=0x7f008c008040 comp rx=0 tx=0).stop 2024-09-15T02:43:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 msgr2=0x7f00980a9560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:43:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0098151130 0x7f00980a9560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:09.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f0078007130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient(hunting): picked mon.c con 0x7f0080003aa0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:09.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0078007a30 0x7f0078009eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient(hunting): picked mon.a con 0x7f0078007a30 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:09.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f007800c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:09.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient(hunting): picked mon.b con 0x7f00980a7150 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:09.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007800d310 con 0x7f0078007a30 2024-09-15T02:43:09.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007800d450 con 0x7f00980a7150 2024-09-15T02:43:09.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:09.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0078015dd0 con 0x7f0080003aa0 2024-09-15T02:43:09.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.484+0000 7f00867fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:09.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f0078007130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): get_auth_request con 0x7f0080003aa0 auth_method 0 2024-09-15T02:43:09.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:09.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f007800c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 10 monclient(hunting): get_auth_request con 0x7f00980a7150 auth_method 0 2024-09-15T02:43:09.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:09.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0095d9b640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0078007a30 0x7f0078009eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:09.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:09.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0095d9b640 10 monclient(hunting): get_auth_request con 0x7f0078007a30 auth_method 0 2024-09-15T02:43:09.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:09.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:09.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0095d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:09.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:09.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:09.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:09.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient(hunting): handle_auth_done global_id 4110 payload 293 2024-09-15T02:43:09.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 msgr2=0x7f007800c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:09.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f007800c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0078007a30 msgr2=0x7f0078009eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:09.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0078007a30 0x7f0078009eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient: _finish_hunting 0 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 monclient: found mon.c 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f00780160c0 con 0x7f0080003aa0 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 10 monclient: _finish_auth 0 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009659c640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f0078007130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f008802bfd0 tx=0x7f0088034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 10 client.4110.objecter ms_handle_connect 0x7f0080003aa0 2024-09-15T02:43:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 10 client.4110.objecter resend_mon_ops 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008802acb0 con 0x7f0080003aa0 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 10 monclient: dump: 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:09.559 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088042070 con 0x7f0080003aa0 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f0088044030 con 0x7f0080003aa0 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.2 v2:172.21.15.53:3301/0 4 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0088047960 con 0x7f0080003aa0 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 10 client.4110.objecter ms_dispatch 0x7f0098000b90 osd_map(1..1 src has 1..1) v4 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f00867fc640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f0087fff640 4 set_mon_vals no callback set 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.485+0000 7f009caef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.486+0000 7f00867fc640 20 client.4110.objecter dump_active .. 0 homeless 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.486+0000 7f009caef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.486+0000 7f009caef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:09.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.486+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0098063ff0 con 0x7f0080003aa0 2024-09-15T02:43:09.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.488+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.2 v2:172.21.15.53:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f008800f040 con 0x7f0080003aa0 2024-09-15T02:43:09.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.488+0000 7f00867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:09.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.488+0000 7f00867fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:09.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.566+0000 7f009caef640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T02:43:09.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.566+0000 7f009caef640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T02:43:09.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.566+0000 7f009caef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:09.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.566+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f00980a66e0 con 0x7f0080003aa0 2024-09-15T02:43:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.569+0000 7f00867fc640 1 -- 172.21.15.53:0/3786999241 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0088000bc0 con 0x7f0080003aa0 2024-09-15T02:43:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.569+0000 7f00867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T02:43:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.569+0000 7f00867fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-15T02:43:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-09-15T02:43:09.572 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:09.571+0000 7f33631bfec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T02:43:09.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 10 librados: watch_flush enter 2024-09-15T02:43:09.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 10 librados: watch_flush exit 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 20 client.4110.objecter shutdown clearing up homeless session... 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 10 client.4110.objecter successfully canceled tick 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 10 monclient: shutdown 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 msgr2=0x7f0078007130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f0078007130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f008802bfd0 tx=0x7f0088034000 comp rx=0 tx=0).stop 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 shutdown_connections 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00980a7150 0x7f007800c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0078007a30 0x7f0078009eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 --2- 172.21.15.53:0/3786999241 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0080003aa0 0x7f0078007130 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 >> 172.21.15.53:0/3786999241 conn(0x7f0098062d90 msgr2=0x7f009813c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 shutdown_connections 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 -- 172.21.15.53:0/3786999241 wait complete. 2024-09-15T02:43:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:09.570+0000 7f009caef640 1 librados: shutdown 2024-09-15T02:43:09.583 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T02:43:09.583 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:09.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T02:43:09.592 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:09.590+0000 7f563fbc1ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T02:43:09.609 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:09.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T02:43:09.610 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:09.609+0000 7f33631bfec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T02:43:09.635 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:09.632+0000 7f563fbc1ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T02:43:09.661 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:09.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T02:43:09.662 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:09.658+0000 7f33631bfec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T02:43:09.688 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:09.686+0000 7f563fbc1ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T02:43:09.713 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:43:09.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T02:43:09.779 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:09.779 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T02:43:09.804 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:09.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T02:43:09.809 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:09.808+0000 7f33631bfec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:43:09.833 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:09.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-15T02:43:09.839 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:09.837+0000 7f563fbc1ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:43:09.898 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T02:43:09.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-15T02:43:09.921 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:09.920+0000 7f33631bfec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T02:43:09.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new ab035ba6-bb97-46de-9069-4082e1295127 0 2024-09-15T02:43:09.951 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:09.949+0000 7f563fbc1ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T02:43:10.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.077+0000 7f524e2c0640 10 monclient: get_monmap_and_config 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.077+0000 7f524e2c0640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.077+0000 7f524e2c0640 10 monclient: monmap: 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:10.078539+0000 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:10.078539+0000 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:10.079 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:10.080 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:10.080 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:10.080 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:10.080 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.078+0000 7f524e2c0640 1 Processor -- start 2024-09-15T02:43:10.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.078+0000 7f524e2c0640 1 -- start start 2024-09-15T02:43:10.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.078+0000 7f524e2c0640 10 monclient: init 2024-09-15T02:43:10.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T02:43:10.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f52480b0e80 0x7f52480b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient(hunting): picked mon.noname-a con 0x7f52480b0e80 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52480b1250 0x7f524813def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f52480b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f524813e430 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524813da60 con 0x7f524813e430 2024-09-15T02:43:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52480b0990 con 0x7f52480b0e80 2024-09-15T02:43:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524813e8b0 con 0x7f52480b1250 2024-09-15T02:43:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524e2c0640 10 monclient(hunting): authenticate will time out at 718.699117s 2024-09-15T02:43:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f52480b0e80 0x7f52480b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52480b1250 0x7f524813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524c85d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f524813e430 0x7f52480af3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.104:50992/0 (socket says 172.21.15.104:50992) 2024-09-15T02:43:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f52480b0e80 0x7f52480b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.104:53782/0 (socket says 172.21.15.104:53782) 2024-09-15T02:43:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f524c85d640 1 -- 172.21.15.104:0/1684463627 learned_addr learned my addr 172.21.15.104:0/1684463627 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-15T02:43:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52480b1250 0x7f524813def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.104:39272/0 (socket says 172.21.15.104:39272) 2024-09-15T02:43:10.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): get_auth_request con 0x7f52480b0e80 auth_method 0 2024-09-15T02:43:10.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 10 monclient(hunting): get_auth_request con 0x7f52480b1250 auth_method 0 2024-09-15T02:43:10.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 1 -- 172.21.15.104:0/1684463627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52480b1250 msgr2=0x7f524813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 1 --2- 172.21.15.104:0/1684463627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52480b1250 0x7f524813def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 1 -- 172.21.15.104:0/1684463627 >> v1:172.21.15.53:6789/0 conn(0x7f524813e430 legacy=0x7f52480af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 1 monclient: found mon.noname-a 2024-09-15T02:43:10.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5247fff640 1 -- 172.21.15.104:0/1684463627 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52480a6a30 con 0x7f52480b0e80 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5246ffd640 1 -- 172.21.15.104:0/1684463627 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756804931 0 0) 0x7f524813da60 con 0x7f524813e430 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.079+0000 7f5246ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5247fff640 1 --2- 172.21.15.104:0/1684463627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b0e80 0x7f52480b0450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5230009880 tx=0x7f52300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ba5aa62643779f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 1 -- 172.21.15.104:0/1684463627 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523002ace0 con 0x7f52480b0e80 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: dump: 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.107 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f524e2c0640 5 monclient: authenticate success, global_id 4113 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 1 -- 172.21.15.104:0/1684463627 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523002ae40 con 0x7f52480b0e80 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f524e2c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:10.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f524e2c0640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f524e2c0640 4 set_mon_vals no callback set 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 1 -- 172.21.15.104:0/1684463627 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5230032670 con 0x7f52480b0e80 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.080+0000 7f5246ffd640 10 monclient: dump: 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.109 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 10 monclient: shutdown 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 -- 172.21.15.104:0/1684463627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b0e80 msgr2=0x7f52480b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 --2- 172.21.15.104:0/1684463627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b0e80 0x7f52480b0450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5230009880 tx=0x7f52300090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 -- 172.21.15.104:0/1684463627 shutdown_connections 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 --2- 172.21.15.104:0/1684463627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52480b1250 0x7f524813def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 --2- 172.21.15.104:0/1684463627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b0e80 0x7f52480b0450 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 -- 172.21.15.104:0/1684463627 >> 172.21.15.104:0/1684463627 conn(0x7f5248062d90 msgr2=0x7f5248063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 -- 172.21.15.104:0/1684463627 shutdown_connections 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 -- 172.21.15.104:0/1684463627 wait complete. 2024-09-15T02:43:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 10 monclient: monmap: 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 librados: starting msgr at 2024-09-15T02:43:10.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.081+0000 7f524e2c0640 1 librados: starting objecter 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 Processor -- start 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 -- start start 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 librados: setting wanted keys 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 librados: calling monclient init 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient: init 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f52480a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f52480b0e80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 0x7f52481507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): picked mon.noname-a con 0x7f52480b1250 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f52480aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): picked mon.noname-c con 0x7f524813e430 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480a71c0 con 0x7f52480b1250 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480abe30 con 0x7f52480b0e80 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5248151240 con 0x7f524813e430 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524e2c0640 10 monclient(hunting): authenticate will time out at 718.702820s 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f5247fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f52480a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f5247fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f52480a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37912/0 (socket says 172.21.15.104:37912) 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f5247fff640 1 -- 172.21.15.104:0/582117192 learned_addr learned my addr 172.21.15.104:0/582117192 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f5247fff640 10 monclient(hunting): get_auth_request con 0x7f52480b0e80 auth_method 0 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 0x7f52481507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f5247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f5247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f5247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.082+0000 7f524c85d640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f52480aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): get_auth_request con 0x7f52480b1250 auth_method 0 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524c85d640 10 monclient(hunting): get_auth_request con 0x7f524813e430 auth_method 0 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524c85d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524c85d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524c85d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524c85d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524c85d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524c85d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient(hunting): handle_auth_done global_id 4119 payload 824 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 msgr2=0x7f52480aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f52480aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 msgr2=0x7f52480a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f52480a9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 monclient: found mon.noname-a 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230009390 con 0x7f52480b1250 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f5247fff640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f52480a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 0x7f52481507c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f523800e960 tx=0x7f52380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 client.?.objecter ms_handle_connect 0x7f52480b1250 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238009540 con 0x7f52480b1250 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: dump: 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.116 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523800c940 con 0x7f52480b1250 2024-09-15T02:43:10.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 5 monclient: authenticate success, global_id 4119 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800ccc0 con 0x7f52480b1250 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 10 monclient: _renew_subs 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52480aaac0 con 0x7f52480b1250 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52467fc640 4 set_mon_vals no callback set 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: dump: 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.118 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 10 monclient: _renew_subs 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52480aaf90 con 0x7f52480b1250 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f524e2c0640 1 librados: init done 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 msgr2=0x7f52481507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 msgr2=0x7f52481507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 0x7f52481507c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f523800e960 tx=0x7f52380090f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 0x7f52481507c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f523800e960 tx=0x7f52380090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 msgr2=0x7f52481507c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52480b1250 0x7f52481507c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f5228007130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 10 monclient(hunting): picked mon.c con 0x7f524813e430 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.083+0000 7f523ffff640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f5228009a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient(hunting): picked mon.b con 0x7f52480b0e80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f522800a3e0 0x7f522800c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient(hunting): picked mon.a con 0x7f522800a3e0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522800d310 con 0x7f522800a3e0 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522800d450 con 0x7f52480b0e80 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522800d5c0 con 0x7f524813e430 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f5228007130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f5228009a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 10 monclient(hunting): get_auth_request con 0x7f524813e430 auth_method 0 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): get_auth_request con 0x7f52480b0e80 auth_method 0 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f522800a3e0 0x7f522800c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 10 monclient(hunting): get_auth_request con 0x7f522800a3e0 auth_method 0 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f524c85d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient(hunting): handle_auth_done global_id 4119 payload 293 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 msgr2=0x7f5228007130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f5228007130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f522800a3e0 msgr2=0x7f522800c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f522800a3e0 0x7f522800c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 monclient: found mon.b 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f522800d8b0 con 0x7f52480b0e80 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5247fff640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f5228007130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 client.4119.objecter ms_handle_connect 0x7f52480b0e80 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 client.4119.objecter resend_mon_ops 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f52477fe640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f5228009a60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5238019000 tx=0x7f523801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238005970 con 0x7f52480b0e80 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient: dump: 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5238013680 con 0x7f52480b0e80 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f5245ffb640 4 set_mon_vals no callback set 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.084+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f52380137f0 con 0x7f52480b0e80 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.1 v2:172.21.15.104:3300/0 4 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5238006390 con 0x7f52480b0e80 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f523ffff640 10 client.4119.objecter ms_dispatch 0x7f5248000b90 osd_map(2..2 src has 1..2) v4 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f523ffff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f523ffff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f523ffff640 20 client.4119.objecter dump_active .. 0 homeless 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f524e2c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f524e2c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f524e2c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.085+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5220005010 con 0x7f52480b0e80 2024-09-15T02:43:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.087+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.1 v2:172.21.15.104:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f5238021060 con 0x7f52480b0e80 2024-09-15T02:43:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.087+0000 7f523ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.087+0000 7f523ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:10.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.151+0000 7f524e2c0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ab035ba6-bb97-46de-9069-4082e1295127", "id": 0}] 2024-09-15T02:43:10.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.151+0000 7f524e2c0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab035ba6-bb97-46de-9069-4082e1295127", "id": 0}] 2024-09-15T02:43:10.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.151+0000 7f524e2c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.151+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ab035ba6-bb97-46de-9069-4082e1295127", "id": 0} v 0) v1 -- 0x7f5220005590 con 0x7f52480b0e80 2024-09-15T02:43:10.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.157+0000 7f523ffff640 1 -- 172.21.15.104:0/582117192 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ab035ba6-bb97-46de-9069-4082e1295127", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5238017070 con 0x7f52480b0e80 2024-09-15T02:43:10.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.157+0000 7f523ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ab035ba6-bb97-46de-9069-4082e1295127", "id": 0}] 2024-09-15T02:43:10.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.157+0000 7f523ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:10.159 INFO:teuthology.orchestra.run.smithi104.stdout:0 2024-09-15T02:43:10.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.157+0000 7f524e2c0640 10 librados: watch_flush enter 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.157+0000 7f524e2c0640 10 librados: watch_flush exit 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.157+0000 7f524e2c0640 20 client.4119.objecter shutdown clearing up homeless session... 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.157+0000 7f524e2c0640 10 client.4119.objecter successfully canceled tick 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 10 monclient: shutdown 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 msgr2=0x7f5228009a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f5228009a60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5238019000 tx=0x7f523801a000 comp rx=0 tx=0).stop 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 shutdown_connections 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f522800a3e0 0x7f522800c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52480b0e80 0x7f5228009a60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 --2- 172.21.15.104:0/582117192 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f524813e430 0x7f5228007130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 >> 172.21.15.104:0/582117192 conn(0x7f5248062d90 msgr2=0x7f5248141fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 shutdown_connections 2024-09-15T02:43:10.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 -- 172.21.15.104:0/582117192 wait complete. 2024-09-15T02:43:10.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.158+0000 7f524e2c0640 1 librados: shutdown 2024-09-15T02:43:10.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new eb614e79-5e04-45a4-9962-84c7619ddb87 1 2024-09-15T02:43:10.235 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:10.234+0000 7f33631bfec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T02:43:10.240 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:10.239+0000 7f563fbc1ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T02:43:10.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.259+0000 7f9cde55d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:10.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.259+0000 7f9cde55d640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.259+0000 7f9cde55d640 10 monclient: monmap: 2024-09-15T02:43:10.261 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.261 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.261 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:10.260611+0000 2024-09-15T02:43:10.261 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:10.260611+0000 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:10.262 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.260+0000 7f9cde55d640 1 Processor -- start 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.260+0000 7f9cde55d640 1 -- start start 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.260+0000 7f9cde55d640 10 monclient: init 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9cd80b0e80 0x7f9cd80b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd80b0e80 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd80b1250 0x7f9cd813def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd80b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9cd813e430 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd813da60 con 0x7f9cd813e430 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd80b0990 con 0x7f9cd80b0e80 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd813e8b0 con 0x7f9cd80b1250 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cde55d640 10 monclient(hunting): authenticate will time out at 718.881155s 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9cd80b0e80 0x7f9cd80b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cdcafa640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9cd813e430 0x7f9cd80af3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:44126/0 (socket says 172.21.15.104:44126) 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cdcafa640 1 -- 172.21.15.104:0/1389046943 learned_addr learned my addr 172.21.15.104:0/1389046943 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): get_auth_request con 0x7f9cd80b0e80 auth_method 0 2024-09-15T02:43:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 1 --2- 172.21.15.104:0/1389046943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd80b1250 0x7f9cd813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 10 monclient(hunting): get_auth_request con 0x7f9cd80b1250 auth_method 0 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 1 -- 172.21.15.104:0/1389046943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd80b1250 msgr2=0x7f9cd813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/1389046943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd80b1250 0x7f9cd813def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 1 -- 172.21.15.104:0/1389046943 >> v1:172.21.15.104:6789/0 conn(0x7f9cd813e430 legacy=0x7f9cd80af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 1 monclient: found mon.noname-c 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd7fff640 1 -- 172.21.15.104:0/1389046943 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd80a6a30 con 0x7f9cd80b0e80 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd6ffd640 1 -- 172.21.15.104:0/1389046943 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378360714 0 0) 0x7f9cd813da60 con 0x7f9cd813e430 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.261+0000 7f9cd6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/1389046943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80b0450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009880 tx=0x7f9cc80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec380bfb7658eda3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 1 -- 172.21.15.104:0/1389046943 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc802ace0 con 0x7f9cd80b0e80 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: dump: 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.282 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cde55d640 5 monclient: authenticate success, global_id 4114 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 1 -- 172.21.15.104:0/1389046943 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc802ae40 con 0x7f9cd80b0e80 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cde55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cde55d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cde55d640 4 set_mon_vals no callback set 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 1 -- 172.21.15.104:0/1389046943 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8032670 con 0x7f9cd80b0e80 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.262+0000 7f9cd6ffd640 10 monclient: dump: 2024-09-15T02:43:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 10 monclient: shutdown 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 -- 172.21.15.104:0/1389046943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 msgr2=0x7f9cd80b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 --2- 172.21.15.104:0/1389046943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80b0450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009880 tx=0x7f9cc80090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 -- 172.21.15.104:0/1389046943 shutdown_connections 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 --2- 172.21.15.104:0/1389046943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd80b1250 0x7f9cd813def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 --2- 172.21.15.104:0/1389046943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80b0450 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 -- 172.21.15.104:0/1389046943 >> 172.21.15.104:0/1389046943 conn(0x7f9cd8062d90 msgr2=0x7f9cd8063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 -- 172.21.15.104:0/1389046943 shutdown_connections 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 1 -- 172.21.15.104:0/1389046943 wait complete. 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.263+0000 7f9cde55d640 10 monclient: monmap: 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 librados: starting msgr at 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 librados: starting objecter 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 Processor -- start 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 -- start start 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 librados: setting wanted keys 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 librados: calling monclient init 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient: init 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd80b0e80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cd81507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): picked mon.noname-a con 0x7f9cd80b1250 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 0x7f9cd80aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd813e430 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd80a71c0 con 0x7f9cd80b1250 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd80abe30 con 0x7f9cd80b0e80 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd8151240 con 0x7f9cd813e430 2024-09-15T02:43:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.264+0000 7f9cde55d640 10 monclient(hunting): authenticate will time out at 718.884981s 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37936/0 (socket says 172.21.15.104:37936) 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 -- 172.21.15.104:0/2954742136 learned_addr learned my addr 172.21.15.104:0/2954742136 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): get_auth_request con 0x7f9cd80b0e80 auth_method 0 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cd81507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 0x7f9cd80aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 10 monclient(hunting): get_auth_request con 0x7f9cd80b1250 auth_method 0 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 10 monclient(hunting): get_auth_request con 0x7f9cd813e430 auth_method 0 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cdcafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 msgr2=0x7f9cd80aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 0x7f9cd80aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 msgr2=0x7f9cd81507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cd81507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 monclient: found mon.noname-b 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc8009390 con 0x7f9cd80b0e80 2024-09-15T02:43:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80a9a40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009880 tx=0x7f9cc8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 client.?.objecter ms_handle_connect 0x7f9cd80b0e80 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc803b6e0 con 0x7f9cd80b0e80 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: dump: 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 5 monclient: authenticate success, global_id 4120 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc803b840 con 0x7f9cd80b0e80 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 10 monclient: _renew_subs 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd80aaac0 con 0x7f9cd80b0e80 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc803bc40 con 0x7f9cd80b0e80 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9ccffff640 10 monclient: dump: 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 10 monclient: _renew_subs 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd67fc640 4 set_mon_vals no callback set 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd80aafb0 con 0x7f9cd80b0e80 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cde55d640 1 librados: init done 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.265+0000 7f9cd7fff640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 msgr2=0x7f9cd80a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 msgr2=0x7f9cd80a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:43:10.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80a9a40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009880 tx=0x7f9cc8032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80a9a40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009880 tx=0x7f9cc8032000 comp rx=0 tx=0).stop 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 msgr2=0x7f9cd80a9a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cd80b0e80 0x7f9cd80a9a40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 0x7f9cb8007130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient(hunting): picked mon.c con 0x7f9cd813e430 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cb8007a30 0x7f9cb8009eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient(hunting): picked mon.b con 0x7f9cb8007a30 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cb800c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient(hunting): picked mon.a con 0x7f9cd80b1250 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb800d350 con 0x7f9cd80b1250 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb800d490 con 0x7f9cb8007a30 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb800d600 con 0x7f9cd813e430 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cb8007a30 0x7f9cb8009eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 10 monclient(hunting): get_auth_request con 0x7f9cb8007a30 auth_method 0 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 0x7f9cb8007130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 10 monclient(hunting): get_auth_request con 0x7f9cd813e430 auth_method 0 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cb800c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): get_auth_request con 0x7f9cd80b1250 auth_method 0 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 msgr2=0x7f9cb8007130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 0x7f9cb8007130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cb8007a30 msgr2=0x7f9cb8009eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cb8007a30 0x7f9cb8009eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 monclient: found mon.a 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f9cb800d8f0 con 0x7f9cd80b1250 2024-09-15T02:43:10.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cd77fe640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cb8007a30 0x7f9cb8009eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cdcafa640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cb800c9d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0001560 tx=0x7f9cd00019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 client.4120.objecter ms_handle_connect 0x7f9cd80b1250 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9ccffff640 10 client.4120.objecter resend_mon_ops 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cde55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cde55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cde55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.266+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd8151400 con 0x7f9cd80b1250 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0004d20 con 0x7f9cd80b1250 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 10 monclient: dump: 2024-09-15T02:43:10.299 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd00075c0 con 0x7f9cd80b1250 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f9cd001c430 con 0x7f9cd80b1250 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.0 v2:172.21.15.53:3300/0 4 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f9cd0007a00 con 0x7f9cd80b1250 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9cd5ffb640 4 set_mon_vals no callback set 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 10 client.4120.objecter ms_dispatch 0x7f9cd8000b90 osd_map(3..3 src has 1..3) v4 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.267+0000 7f9ccffff640 20 client.4120.objecter dump_active .. 0 homeless 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.269+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.0 v2:172.21.15.53:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f9cd001c6e0 con 0x7f9cd80b1250 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.269+0000 7f9ccffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.269+0000 7f9ccffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:10.302 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:10.299+0000 7f33631bfec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T02:43:10.302 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:10.290+0000 7f563fbc1ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T02:43:10.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.335+0000 7f9cde55d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eb614e79-5e04-45a4-9962-84c7619ddb87", "id": 1}] 2024-09-15T02:43:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.335+0000 7f9cde55d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eb614e79-5e04-45a4-9962-84c7619ddb87", "id": 1}] 2024-09-15T02:43:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.335+0000 7f9cde55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.335+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "eb614e79-5e04-45a4-9962-84c7619ddb87", "id": 1} v 0) v1 -- 0x7f9cd8063ff0 con 0x7f9cd80b1250 2024-09-15T02:43:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.338+0000 7f9ccffff640 1 -- 172.21.15.104:0/2954742136 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb614e79-5e04-45a4-9962-84c7619ddb87", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9cd001ca90 con 0x7f9cd80b1250 2024-09-15T02:43:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.338+0000 7f9ccffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eb614e79-5e04-45a4-9962-84c7619ddb87", "id": 1}] 2024-09-15T02:43:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.338+0000 7f9ccffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:10.339 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.338+0000 7f9cde55d640 10 librados: watch_flush enter 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.338+0000 7f9cde55d640 10 librados: watch_flush exit 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.338+0000 7f9cde55d640 20 client.4120.objecter shutdown clearing up homeless session... 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.338+0000 7f9cde55d640 10 client.4120.objecter successfully canceled tick 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 10 monclient: shutdown 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 msgr2=0x7f9cb800c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cb800c9d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0001560 tx=0x7f9cd00019a0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 shutdown_connections 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd80b1250 0x7f9cb800c9d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9cb8007a30 0x7f9cb8009eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 --2- 172.21.15.104:0/2954742136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9cd813e430 0x7f9cb8007130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 >> 172.21.15.104:0/2954742136 conn(0x7f9cd8062d90 msgr2=0x7f9cd8141fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 shutdown_connections 2024-09-15T02:43:10.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 -- 172.21.15.104:0/2954742136 wait complete. 2024-09-15T02:43:10.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.339+0000 7f9cde55d640 1 librados: shutdown 2024-09-15T02:43:10.350 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 08ea49a0-28e1-4740-a4b4-d722560b15c2 2 2024-09-15T02:43:10.431 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:10.428+0000 7f563fbc1ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T02:43:10.438 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:10.437+0000 7f33631bfec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T02:43:10.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.441+0000 7efe4051d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:10.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.441+0000 7efe4051d640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.441+0000 7efe4051d640 10 monclient: monmap: 2024-09-15T02:43:10.443 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.443 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:10.442758+0000 2024-09-15T02:43:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:10.442758+0000 2024-09-15T02:43:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:10.445 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.442+0000 7efe4051d640 1 Processor -- start 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.442+0000 7efe4051d640 1 -- start start 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.442+0000 7efe4051d640 10 monclient: init 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe3813a800 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3806b870 0x7efe3806c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient(hunting): picked mon.noname-b con 0x7efe3806b870 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efe3806ea90 0x7efe3806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient(hunting): picked mon.noname-a con 0x7efe3806ea90 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe38138240 con 0x7efe3813a800 2024-09-15T02:43:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe3806bfd0 con 0x7efe3806ea90 2024-09-15T02:43:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3813abd0 con 0x7efe3806b870 2024-09-15T02:43:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe4051d640 10 monclient(hunting): authenticate will time out at 719.063319s 2024-09-15T02:43:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efe3806ea90 0x7efe3806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3806b870 0x7efe3806c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efe3806ea90 0x7efe3806eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.104:53824/0 (socket says 172.21.15.104:53824) 2024-09-15T02:43:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 -- 172.21.15.104:0/3092702611 learned_addr learned my addr 172.21.15.104:0/3092702611 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 10 monclient(hunting): get_auth_request con 0x7efe3806b870 auth_method 0 2024-09-15T02:43:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3e2b9640 1 --1- 172.21.15.104:0/3092702611 >> v1:172.21.15.53:6789/0 conn(0x7efe3813a800 0x7efe3806e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.104:50998/0 (socket says 172.21.15.104:50998) 2024-09-15T02:43:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): get_auth_request con 0x7efe3806ea90 auth_method 0 2024-09-15T02:43:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-09-15T02:43:10.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 -- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3806b870 msgr2=0x7efe3806c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:10.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 --2- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3806b870 0x7efe3806c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 -- 172.21.15.104:0/3092702611 >> v1:172.21.15.53:6789/0 conn(0x7efe3813a800 legacy=0x7efe3806e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 monclient: found mon.noname-a 2024-09-15T02:43:10.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3d2b7640 1 -- 172.21.15.104:0/3092702611 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977411699 0 0) 0x7efe38138240 con 0x7efe3813a800 2024-09-15T02:43:10.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3eaba640 1 -- 172.21.15.104:0/3092702611 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3806d900 con 0x7efe3806ea90 2024-09-15T02:43:10.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3d2b7640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:10.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.443+0000 7efe3dab8640 1 --2- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3806b870 0x7efe3806c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:10.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3eaba640 1 --2- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806ea90 0x7efe3806eed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe30009f20 tx=0x7efe30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce78b0237108377 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 1 -- 172.21.15.104:0/3092702611 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe30031400 con 0x7efe3806ea90 2024-09-15T02:43:10.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:10.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: dump: 2024-09-15T02:43:10.471 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.471 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.471 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.472 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.472 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.472 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.472 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.472 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.473 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe4051d640 5 monclient: authenticate success, global_id 4125 2024-09-15T02:43:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 1 -- 172.21.15.104:0/3092702611 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3002d050 con 0x7efe3806ea90 2024-09-15T02:43:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe4051d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe4051d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe4051d640 4 set_mon_vals no callback set 2024-09-15T02:43:10.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 1 -- 172.21.15.104:0/3092702611 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe30031790 con 0x7efe3806ea90 2024-09-15T02:43:10.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:10.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.444+0000 7efe3d2b7640 10 monclient: dump: 2024-09-15T02:43:10.475 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.475 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.476 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.476 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.476 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.476 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.476 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.477 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.477 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.477 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 10 monclient: shutdown 2024-09-15T02:43:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 -- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806ea90 msgr2=0x7efe3806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 --2- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806ea90 0x7efe3806eed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe30009f20 tx=0x7efe30009840 comp rx=0 tx=0).stop 2024-09-15T02:43:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 -- 172.21.15.104:0/3092702611 shutdown_connections 2024-09-15T02:43:10.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 --2- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806ea90 0x7efe3806eed0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 --2- 172.21.15.104:0/3092702611 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3806b870 0x7efe3806c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 -- 172.21.15.104:0/3092702611 >> 172.21.15.104:0/3092702611 conn(0x7efe380b1af0 msgr2=0x7efe380681f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 -- 172.21.15.104:0/3092702611 shutdown_connections 2024-09-15T02:43:10.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.445+0000 7efe4051d640 1 -- 172.21.15.104:0/3092702611 wait complete. 2024-09-15T02:43:10.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 10 monclient: monmap: 2024-09-15T02:43:10.482 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.482 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:10.482 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:10.482 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:10.483 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.483 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.483 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:10.483 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:10.483 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:10.483 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 1 librados: starting msgr at 2024-09-15T02:43:10.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 1 librados: starting objecter 2024-09-15T02:43:10.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 1 Processor -- start 2024-09-15T02:43:10.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 1 -- start start 2024-09-15T02:43:10.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 1 librados: setting wanted keys 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 1 librados: calling monclient init 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.446+0000 7efe4051d640 10 monclient: init 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806b870 0x7efe380ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): picked mon.noname-a con 0x7efe3806b870 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe3806ea90 0x7efe380a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): picked mon.noname-b con 0x7efe3806ea90 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3813a800 0x7efe38150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): picked mon.noname-c con 0x7efe3813a800 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3811a790 con 0x7efe3806b870 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3811a990 con 0x7efe3806ea90 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe38118d00 con 0x7efe3813a800 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient(hunting): authenticate will time out at 719.067150s 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe3806ea90 0x7efe380a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe3806ea90 0x7efe380a9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37954/0 (socket says 172.21.15.104:37954) 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 -- 172.21.15.104:0/3229806132 learned_addr learned my addr 172.21.15.104:0/3229806132 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): get_auth_request con 0x7efe3806ea90 auth_method 0 2024-09-15T02:43:10.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806b870 0x7efe380ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3813a800 0x7efe38150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 10 monclient(hunting): get_auth_request con 0x7efe3806b870 auth_method 0 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 10 monclient(hunting): get_auth_request con 0x7efe3813a800 auth_method 0 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3e2b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3eaba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 -- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3813a800 msgr2=0x7efe38150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3813a800 0x7efe38150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 -- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806b870 msgr2=0x7efe380ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806b870 0x7efe380ac610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 monclient: found mon.noname-b 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 -- 172.21.15.104:0/3229806132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3400b630 con 0x7efe3806ea90 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3dab8640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe3806ea90 0x7efe380a9ac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efe3400ede0 tx=0x7efe3400b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 client.?.objecter ms_handle_connect 0x7efe3806ea90 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 1 -- 172.21.15.104:0/3229806132 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34010440 con 0x7efe3806ea90 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: dump: 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.489 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 5 monclient: authenticate success, global_id 4123 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient: _renew_subs 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe30009ae0 con 0x7efe3806ea90 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 1 -- 172.21.15.104:0/3229806132 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe340105a0 con 0x7efe3806ea90 2024-09-15T02:43:10.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 1 -- 172.21.15.104:0/3229806132 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34010920 con 0x7efe3806ea90 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe2e7fc640 10 monclient: dump: 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.491 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe3cab6640 4 set_mon_vals no callback set 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient: _renew_subs 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe38119740 con 0x7efe3806ea90 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.447+0000 7efe4051d640 1 librados: init done 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.448+0000 7efe2e7fc640 1 -- 172.21.15.104:0/3229806132 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7efe34005970 con 0x7efe3806ea90 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.448+0000 7efe2e7fc640 1 -- 172.21.15.104:0/3229806132 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7efe34006540 con 0x7efe3806ea90 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.448+0000 7efe2e7fc640 10 client.4123.objecter ms_dispatch 0x7efe38000b90 osd_map(4..4 src has 1..4) v4 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.448+0000 7efe2e7fc640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.448+0000 7efe2e7fc640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.448+0000 7efe2e7fc640 20 client.4123.objecter dump_active .. 0 homeless 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.449+0000 7efe4051d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.449+0000 7efe4051d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.449+0000 7efe4051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.449+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe38118ec0 con 0x7efe3806ea90 2024-09-15T02:43:10.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.451+0000 7efe2e7fc640 1 -- 172.21.15.104:0/3229806132 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7efe3401d020 con 0x7efe3806ea90 2024-09-15T02:43:10.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.451+0000 7efe2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.451+0000 7efe2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.535+0000 7efe4051d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "08ea49a0-28e1-4740-a4b4-d722560b15c2", "id": 2}] 2024-09-15T02:43:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.535+0000 7efe4051d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "08ea49a0-28e1-4740-a4b4-d722560b15c2", "id": 2}] 2024-09-15T02:43:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.535+0000 7efe4051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.535+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd new", "uuid": "08ea49a0-28e1-4740-a4b4-d722560b15c2", "id": 2} v 0) v1 -- 0x7efe38063ff0 con 0x7efe3806ea90 2024-09-15T02:43:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe2e7fc640 1 -- 172.21.15.104:0/3229806132 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "08ea49a0-28e1-4740-a4b4-d722560b15c2", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7efe34005e00 con 0x7efe3806ea90 2024-09-15T02:43:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "08ea49a0-28e1-4740-a4b4-d722560b15c2", "id": 2}] 2024-09-15T02:43:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe2e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:10.544 INFO:teuthology.orchestra.run.smithi104.stdout:2 2024-09-15T02:43:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 10 librados: watch_flush enter 2024-09-15T02:43:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 10 librados: watch_flush exit 2024-09-15T02:43:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 20 client.4123.objecter shutdown clearing up homeless session... 2024-09-15T02:43:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 10 client.4123.objecter successfully canceled tick 2024-09-15T02:43:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 10 monclient: shutdown 2024-09-15T02:43:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe3806ea90 msgr2=0x7efe380a9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe3806ea90 0x7efe380a9ac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efe3400ede0 tx=0x7efe3400b3b0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 shutdown_connections 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe3813a800 0x7efe38150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe3806ea90 0x7efe380a9ac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 --2- 172.21.15.104:0/3229806132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe3806b870 0x7efe380ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 >> 172.21.15.104:0/3229806132 conn(0x7efe380b1af0 msgr2=0x7efe3806df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.543+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 shutdown_connections 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.544+0000 7efe4051d640 1 -- 172.21.15.104:0/3229806132 wait complete. 2024-09-15T02:43:10.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.544+0000 7efe4051d640 1 librados: shutdown 2024-09-15T02:43:10.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 1a3a89c3-b449-46ba-bc65-3f53371d3af1 3 2024-09-15T02:43:10.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.646+0000 7f2a9446f640 10 monclient: get_monmap_and_config 2024-09-15T02:43:10.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.646+0000 7f2a9446f640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.646+0000 7f2a9446f640 10 monclient: monmap: 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:10.647295+0000 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:10.647295+0000 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:10.648 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.646+0000 7f2a9446f640 1 Processor -- start 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 1 -- start start 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient: init 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b0e80 0x7f2a8c0b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient(hunting): picked mon.noname-b con 0x7f2a8c0b0e80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a8c0b1250 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2a8c13e600 0x7f2a8c0af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient(hunting): picked mon.noname-a con 0x7f2a8c13e600 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c13da60 con 0x7f2a8c0b1250 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c0b0990 con 0x7f2a8c13e600 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0a6a30 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a9446f640 10 monclient(hunting): authenticate will time out at 719.267883s 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.647+0000 7f2a91a0a640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2a8c0b1250 0x7f2a8c13def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:44142/0 (socket says 172.21.15.104:44142) 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a91a0a640 1 -- 172.21.15.104:0/3223834287 learned_addr learned my addr 172.21.15.104:0/3223834287 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 1 --2- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b0e80 0x7f2a8c0b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 1 --2- 172.21.15.104:0/3223834287 >> v2:172.21.15.53:3300/0 conn(0x7f2a8c13e600 0x7f2a8c0af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 10 monclient(hunting): get_auth_request con 0x7f2a8c0b0e80 auth_method 0 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): get_auth_request con 0x7f2a8c13e600 auth_method 0 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a9220b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a91209640 1 -- 172.21.15.104:0/3223834287 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120012183 0 0) 0x7f2a8c13da60 con 0x7f2a8c0b1250 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a91209640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a91209640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a91209640 10 monclient(hunting): my global_id is 4126 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a91209640 1 -- 172.21.15.104:0/3223834287 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a780034e0 con 0x7f2a8c0b1250 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient(hunting): handle_auth_done global_id 4134 payload 274 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 1 -- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b0e80 msgr2=0x7f2a8c0b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 1 --2- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b0e80 0x7f2a8c0b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 1 -- 172.21.15.104:0/3223834287 >> v1:172.21.15.104:6789/0 conn(0x7f2a8c0b1250 legacy=0x7f2a8c13def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 1 monclient: found mon.noname-a 2024-09-15T02:43:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.648+0000 7f2a92a0c640 1 -- 172.21.15.104:0/3223834287 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c0a5a00 con 0x7f2a8c13e600 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a92a0c640 1 --2- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 0x7f2a8c0af3c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009e90 tx=0x7f2a84009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8feff98668105391 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 1 -- 172.21.15.104:0/3223834287 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8402aea0 con 0x7f2a8c13e600 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: dump: 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 1 -- 172.21.15.104:0/3223834287 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8402d050 con 0x7f2a8c13e600 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 5 monclient: authenticate success, global_id 4134 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 1 -- 172.21.15.104:0/3223834287 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84007780 con 0x7f2a8c13e600 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 4 set_mon_vals no callback set 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a91209640 10 monclient: dump: 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.663 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 10 monclient: shutdown 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 1 -- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 msgr2=0x7f2a8c0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.649+0000 7f2a9446f640 1 --2- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 0x7f2a8c0af3c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009e90 tx=0x7f2a84009780 comp rx=0 tx=0).stop 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 -- 172.21.15.104:0/3223834287 shutdown_connections 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 --2- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 0x7f2a8c0af3c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 --2- 172.21.15.104:0/3223834287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b0e80 0x7f2a8c0b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 -- 172.21.15.104:0/3223834287 >> 172.21.15.104:0/3223834287 conn(0x7f2a8c062d90 msgr2=0x7f2a8c063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 -- 172.21.15.104:0/3223834287 shutdown_connections 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 -- 172.21.15.104:0/3223834287 wait complete. 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 10 monclient: monmap: 2024-09-15T02:43:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 librados: starting msgr at 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.650+0000 7f2a9446f640 1 librados: starting objecter 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 Processor -- start 2024-09-15T02:43:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 -- start start 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 librados: setting wanted keys 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 librados: calling monclient init 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient: init 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0b0e80 0x7f2a8c0aa110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): picked mon.noname-b con 0x7f2a8c0b0e80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b1250 0x7f2a8c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): picked mon.noname-c con 0x7f2a8c0b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 0x7f2a8c0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): picked mon.noname-a con 0x7f2a8c13e600 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0abe10 con 0x7f2a8c13e600 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0ac010 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0ac2f0 con 0x7f2a8c0b1250 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9446f640 10 monclient(hunting): authenticate will time out at 719.271630s 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0b0e80 0x7f2a8c0aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0b0e80 0x7f2a8c0aa110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37958/0 (socket says 172.21.15.104:37958) 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 1 -- 172.21.15.104:0/3601173940 learned_addr learned my addr 172.21.15.104:0/3601173940 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 10 monclient(hunting): get_auth_request con 0x7f2a8c0b0e80 auth_method 0 2024-09-15T02:43:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b1250 0x7f2a8c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 0x7f2a8c0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 10 monclient(hunting): get_auth_request con 0x7f2a8c0b1250 auth_method 0 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 10 monclient(hunting): get_auth_request con 0x7f2a8c13e600 auth_method 0 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a9220b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a91a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.651+0000 7f2a92a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 1 -- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b1250 msgr2=0x7f2a8c0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b1250 0x7f2a8c0acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 1 -- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 msgr2=0x7f2a8c0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 0x7f2a8c0ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 1 monclient: found mon.noname-b 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 1 -- 172.21.15.104:0/3601173940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84009a20 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9220b640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0b0e80 0x7f2a8c0aa110 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e900 tx=0x7f2a880090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 client.?.objecter ms_handle_connect 0x7f2a8c0b0e80 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 1 -- 172.21.15.104:0/3601173940 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800c940 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: dump: 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 1 -- 172.21.15.104:0/3601173940 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8800cac0 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 5 monclient: authenticate success, global_id 4129 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 10 monclient: _renew_subs 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c0ac5d0 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 1 -- 172.21.15.104:0/3601173940 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800ce80 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 monclient: dump: 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a90a08640 4 set_mon_vals no callback set 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 10 monclient: _renew_subs 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c0ab4f0 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a9446f640 1 librados: init done 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 1 -- 172.21.15.104:0/3601173940 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f2a88005970 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 1 -- 172.21.15.104:0/3601173940 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f2a88013dd0 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 10 client.4129.objecter ms_dispatch 0x7f2a8c000b90 osd_map(5..5 src has 1..5) v4 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.652+0000 7f2a827fc640 20 client.4129.objecter dump_active .. 0 homeless 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.653+0000 7f2a9446f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.653+0000 7f2a9446f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.653+0000 7f2a9446f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.653+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c070fe0 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.655+0000 7f2a827fc640 1 -- 172.21.15.104:0/3601173940 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f2a8c070fe0 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.655+0000 7f2a827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.655+0000 7f2a827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:10.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.738+0000 7f2a9446f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1a3a89c3-b449-46ba-bc65-3f53371d3af1", "id": 3}] 2024-09-15T02:43:10.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.738+0000 7f2a9446f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1a3a89c3-b449-46ba-bc65-3f53371d3af1", "id": 3}] 2024-09-15T02:43:10.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.738+0000 7f2a9446f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.738+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1a3a89c3-b449-46ba-bc65-3f53371d3af1", "id": 3} v 0) v1 -- 0x7f2a8c063ff0 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.746+0000 7f2a827fc640 1 -- 172.21.15.104:0/3601173940 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a3a89c3-b449-46ba-bc65-3f53371d3af1", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2a88021060 con 0x7f2a8c0b0e80 2024-09-15T02:43:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.746+0000 7f2a827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1a3a89c3-b449-46ba-bc65-3f53371d3af1", "id": 3}] 2024-09-15T02:43:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.746+0000 7f2a827fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:10.747 INFO:teuthology.orchestra.run.smithi104.stdout:3 2024-09-15T02:43:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.746+0000 7f2a9446f640 10 librados: watch_flush enter 2024-09-15T02:43:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.746+0000 7f2a9446f640 10 librados: watch_flush exit 2024-09-15T02:43:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.746+0000 7f2a9446f640 20 client.4129.objecter shutdown clearing up homeless session... 2024-09-15T02:43:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.746+0000 7f2a9446f640 10 client.4129.objecter successfully canceled tick 2024-09-15T02:43:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 10 monclient: shutdown 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0b0e80 msgr2=0x7f2a8c0aa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0b0e80 0x7f2a8c0aa110 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e900 tx=0x7f2a880090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 shutdown_connections 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a8c13e600 0x7f2a8c0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a8c0b1250 0x7f2a8c0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 --2- 172.21.15.104:0/3601173940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0b0e80 0x7f2a8c0aa110 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 >> 172.21.15.104:0/3601173940 conn(0x7f2a8c062d90 msgr2=0x7f2a8c0707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 shutdown_connections 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 -- 172.21.15.104:0/3601173940 wait complete. 2024-09-15T02:43:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.747+0000 7f2a9446f640 1 librados: shutdown 2024-09-15T02:43:10.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new d969fe78-c8c8-4c03-a534-2173406b0ba6 4 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.849+0000 7fdbb6f9d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.849+0000 7fdbb6f9d640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.849+0000 7fdbb6f9d640 10 monclient: monmap: 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:10.850592+0000 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:10.850592+0000 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:10.852 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:10.852 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:10.852 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:10.852 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.850+0000 7fdbb6f9d640 1 Processor -- start 2024-09-15T02:43:10.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.850+0000 7fdbb6f9d640 1 -- start start 2024-09-15T02:43:10.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.850+0000 7fdbb6f9d640 10 monclient: init 2024-09-15T02:43:10.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb00b0e80 0x7fdbb00b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb00b0e80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdbb00b1250 0x7fdbb013def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb00b1250 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbb013e430 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb013da60 con 0x7fdbb013e430 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00b0990 con 0x7fdbb00b1250 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb013e8b0 con 0x7fdbb00b0e80 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb6f9d640 10 monclient(hunting): authenticate will time out at 719.471104s 2024-09-15T02:43:10.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdbb00b1250 0x7fdbb013def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb553a640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdbb013e430 0x7fdbb00af3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:44158/0 (socket says 172.21.15.104:44158) 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb553a640 1 -- 172.21.15.104:0/2361605477 learned_addr learned my addr 172.21.15.104:0/2361605477 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): get_auth_request con 0x7fdbb00b1250 auth_method 0 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 1 --2- 172.21.15.104:0/2361605477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb00b0e80 0x7fdbb00b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 10 monclient(hunting): get_auth_request con 0x7fdbb00b0e80 auth_method 0 2024-09-15T02:43:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbb4d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 1 -- 172.21.15.104:0/2361605477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb00b0e80 msgr2=0x7fdbb00b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 1 --2- 172.21.15.104:0/2361605477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb00b0e80 0x7fdbb00b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 1 -- 172.21.15.104:0/2361605477 >> v1:172.21.15.104:6789/0 conn(0x7fdbb013e430 legacy=0x7fdbb00af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 1 monclient: found mon.noname-c 2024-09-15T02:43:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaf7fe640 1 -- 172.21.15.104:0/2361605477 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971256376 0 0) 0x7fdbb013da60 con 0x7fdbb013e430 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaffff640 1 -- 172.21.15.104:0/2361605477 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00a6a30 con 0x7fdbb00b1250 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.851+0000 7fdbaf7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaffff640 1 --2- 172.21.15.104:0/2361605477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb013def0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009880 tx=0x7fdb9c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bccd0845fe830516 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 1 -- 172.21.15.104:0/2361605477 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c02ace0 con 0x7fdbb00b1250 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: dump: 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.864 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbb6f9d640 5 monclient: authenticate success, global_id 4132 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbb6f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 1 -- 172.21.15.104:0/2361605477 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c02ae40 con 0x7fdbb00b1250 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 1 -- 172.21.15.104:0/2361605477 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c032670 con 0x7fdbb00b1250 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbb6f9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbb6f9d640 4 set_mon_vals no callback set 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.852+0000 7fdbaf7fe640 10 monclient: dump: 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 10 monclient: shutdown 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2361605477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 msgr2=0x7fdbb013def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 --2- 172.21.15.104:0/2361605477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb013def0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009880 tx=0x7fdb9c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2361605477 shutdown_connections 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 --2- 172.21.15.104:0/2361605477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb013def0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 --2- 172.21.15.104:0/2361605477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb00b0e80 0x7fdbb00b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2361605477 >> 172.21.15.104:0/2361605477 conn(0x7fdbb0062d90 msgr2=0x7fdbb0063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2361605477 shutdown_connections 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2361605477 wait complete. 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 10 monclient: build_initial_monmap 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.853+0000 7fdbb6f9d640 10 monclient: monmap: 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.854+0000 7fdbb6f9d640 1 librados: starting msgr at 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.854+0000 7fdbb6f9d640 1 librados: starting objecter 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.854+0000 7fdbb6f9d640 1 Processor -- start 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.854+0000 7fdbb6f9d640 1 -- start start 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.854+0000 7fdbb6f9d640 1 librados: setting wanted keys 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.854+0000 7fdbb6f9d640 1 librados: calling monclient init 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.854+0000 7fdbb6f9d640 10 monclient: init 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbb00b0e80 0x7fdbb01411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb00b0e80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb00adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb00b1250 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 0x7fdbb00a76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb013e430 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00ab0c0 con 0x7fdbb00b0e80 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00a80c0 con 0x7fdbb00b1250 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb01397c0 con 0x7fdbb013e430 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient(hunting): authenticate will time out at 719.475148s 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb00adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb00adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37980/0 (socket says 172.21.15.104:37980) 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 -- 172.21.15.104:0/2173382948 learned_addr learned my addr 172.21.15.104:0/2173382948 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): get_auth_request con 0x7fdbb00b1250 auth_method 0 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb553a640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 0x7fdbb00a76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbb00b0e80 0x7fdbb01411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb553a640 1 -- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 msgr2=0x7fdbb00a76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb553a640 1 -- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 msgr2=0x7fdbb00a76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 10 monclient(hunting): get_auth_request con 0x7fdbb00b0e80 auth_method 0 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb553a640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 0x7fdbb00a76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb553a640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 0x7fdbb00a76d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb4d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-15T02:43:10.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 -- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 msgr2=0x7fdbb00a76d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 0x7fdbb00a76d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 -- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbb00b0e80 msgr2=0x7fdbb01411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbb00b0e80 0x7fdbb01411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 monclient: found mon.noname-b 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 -- 172.21.15.104:0/2173382948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c009390 con 0x7fdbb00b1250 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaffff640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb00adf70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009880 tx=0x7fdb9c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 client.?.objecter ms_handle_connect 0x7fdbb00b1250 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:10.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 1 -- 172.21.15.104:0/2173382948 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c031780 con 0x7fdbb00b1250 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: dump: 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: _finish_auth 0 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 1 -- 172.21.15.104:0/2173382948 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c031900 con 0x7fdbb00b1250 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 5 monclient: authenticate success, global_id 4138 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient: _renew_subs 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 1 -- 172.21.15.104:0/2173382948 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c031cc0 con 0x7fdbb00b1250 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0141710 con 0x7fdbb00b1250 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbacff9640 10 monclient: dump: 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbaeffd640 4 set_mon_vals no callback set 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient: _renew_subs 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb00a7c10 con 0x7fdbb00b1250 2024-09-15T02:43:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.855+0000 7fdbb6f9d640 1 librados: init done 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbacff9640 1 -- 172.21.15.104:0/2173382948 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fdb9c039040 con 0x7fdbb00b1250 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbacff9640 1 -- 172.21.15.104:0/2173382948 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fdb9c031e20 con 0x7fdbb00b1250 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbacff9640 10 client.4138.objecter ms_dispatch 0x7fdbb0000b90 osd_map(6..6 src has 1..6) v4 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbacff9640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbacff9640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbacff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbb6f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbb6f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbb6f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.856+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb00a8cb0 con 0x7fdbb00b1250 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.858+0000 7fdbacff9640 1 -- 172.21.15.104:0/2173382948 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fdbb00a8cb0 con 0x7fdbb00b1250 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.858+0000 7fdbacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.858+0000 7fdbacff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.925+0000 7fdbb6f9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d969fe78-c8c8-4c03-a534-2173406b0ba6", "id": 4}] 2024-09-15T02:43:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.925+0000 7fdbb6f9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d969fe78-c8c8-4c03-a534-2173406b0ba6", "id": 4}] 2024-09-15T02:43:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.925+0000 7fdbb6f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.925+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d969fe78-c8c8-4c03-a534-2173406b0ba6", "id": 4} v 0) v1 -- 0x7fdbb0063ff0 con 0x7fdbb00b1250 2024-09-15T02:43:10.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.933+0000 7fdbacff9640 1 -- 172.21.15.104:0/2173382948 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d969fe78-c8c8-4c03-a534-2173406b0ba6", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdb9c049060 con 0x7fdbb00b1250 2024-09-15T02:43:10.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.933+0000 7fdbacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d969fe78-c8c8-4c03-a534-2173406b0ba6", "id": 4}] 2024-09-15T02:43:10.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.933+0000 7fdbacff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:10.935 INFO:teuthology.orchestra.run.smithi104.stdout:4 2024-09-15T02:43:10.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 10 librados: watch_flush enter 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 10 librados: watch_flush exit 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 20 client.4138.objecter shutdown clearing up homeless session... 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 10 client.4138.objecter successfully canceled tick 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 10 monclient: shutdown 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 msgr2=0x7fdbb00adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb00adf70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009880 tx=0x7fdb9c032000 comp rx=0 tx=0).stop 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 shutdown_connections 2024-09-15T02:43:10.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbb013e430 0x7fdbb00a76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbb00b1250 0x7fdbb00adf70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 --2- 172.21.15.104:0/2173382948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbb00b0e80 0x7fdbb01411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:10.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 >> 172.21.15.104:0/2173382948 conn(0x7fdbb0062d90 msgr2=0x7fdbb0063160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:10.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 shutdown_connections 2024-09-15T02:43:10.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 -- 172.21.15.104:0/2173382948 wait complete. 2024-09-15T02:43:10.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:10.934+0000 7fdbb6f9d640 1 librados: shutdown 2024-09-15T02:43:10.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 51e17a2f-42d3-49a6-b71e-dc7e13877c2e 5 2024-09-15T02:43:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.047+0000 7f6fefe57640 10 monclient: get_monmap_and_config 2024-09-15T02:43:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.047+0000 7f6fefe57640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.048+0000 7f6fefe57640 10 monclient: monmap: 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:11.048926+0000 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:11.048926+0000 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:11.051 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.048+0000 7f6fefe57640 1 Processor -- start 2024-09-15T02:43:11.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.048+0000 7f6fefe57640 1 -- start start 2024-09-15T02:43:11.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.048+0000 7f6fefe57640 10 monclient: init 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6fe80b0e80 0x7f6fe80b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe80b0e80 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b1250 0x7f6fe813def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe80b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fe813e430 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe813da60 con 0x7f6fe813e430 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe80b0990 con 0x7f6fe80b0e80 2024-09-15T02:43:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe813e8b0 con 0x7f6fe80b1250 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fefe57640 10 monclient(hunting): authenticate will time out at 719.669455s 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b1250 0x7f6fe813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b1250 0x7f6fe813def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.104:39360/0 (socket says 172.21.15.104:39360) 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6fe80b0e80 0x7f6fe80b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 1 -- 172.21.15.104:0/579864777 learned_addr learned my addr 172.21.15.104:0/579864777 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 10 monclient(hunting): get_auth_request con 0x7f6fe80b1250 auth_method 0 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fee3f4640 1 --1- 172.21.15.104:0/579864777 >> v1:172.21.15.53:6789/0 conn(0x7f6fe813e430 0x7f6fe80af3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.104:51000/0 (socket says 172.21.15.104:51000) 2024-09-15T02:43:11.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 10 monclient(hunting): get_auth_request con 0x7f6fe80b0e80 auth_method 0 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fed3f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.049+0000 7f6fedbf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fecbf1640 1 -- 172.21.15.104:0/579864777 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833478143 0 0) 0x7f6fe813da60 con 0x7f6fe813e430 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 1 -- 172.21.15.104:0/579864777 >> v2:172.21.15.53:3300/0 conn(0x7f6fe80b0e80 msgr2=0x7f6fe80b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:11.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 1 --2- 172.21.15.104:0/579864777 >> v2:172.21.15.53:3300/0 conn(0x7f6fe80b0e80 0x7f6fe80b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 1 -- 172.21.15.104:0/579864777 >> v1:172.21.15.53:6789/0 conn(0x7f6fe813e430 legacy=0x7f6fe80af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 1 monclient: found mon.noname-b 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 1 -- 172.21.15.104:0/579864777 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe80a6a30 con 0x7f6fe80b1250 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fecbf1640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fedbf3640 1 --2- 172.21.15.104:0/579864777 >> v2:172.21.15.53:3300/0 conn(0x7f6fe80b0e80 0x7f6fe80b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.050+0000 7f6fed3f2640 1 --2- 172.21.15.104:0/579864777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b1250 0x7f6fe813def0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009910 tx=0x7f6fdc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36bccd37ef97b5a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 1 -- 172.21.15.104:0/579864777 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc02add0 con 0x7f6fe80b1250 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:11.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: dump: 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 1 -- 172.21.15.104:0/579864777 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc032440 con 0x7f6fe80b1250 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 5 monclient: authenticate success, global_id 4136 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 1 -- 172.21.15.104:0/579864777 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0327e0 con 0x7f6fe80b1250 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 4 set_mon_vals no callback set 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fecbf1640 10 monclient: dump: 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.065 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 10 monclient: shutdown 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 -- 172.21.15.104:0/579864777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b1250 msgr2=0x7f6fe813def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 --2- 172.21.15.104:0/579864777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b1250 0x7f6fe813def0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009910 tx=0x7f6fdc0091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 -- 172.21.15.104:0/579864777 shutdown_connections 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 --2- 172.21.15.104:0/579864777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b1250 0x7f6fe813def0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 --2- 172.21.15.104:0/579864777 >> v2:172.21.15.53:3300/0 conn(0x7f6fe80b0e80 0x7f6fe80b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 -- 172.21.15.104:0/579864777 >> 172.21.15.104:0/579864777 conn(0x7f6fe8062d90 msgr2=0x7f6fe8063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 -- 172.21.15.104:0/579864777 shutdown_connections 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.051+0000 7f6fefe57640 1 -- 172.21.15.104:0/579864777 wait complete. 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 10 monclient: monmap: 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:11.067 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 1 librados: starting msgr at 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 1 librados: starting objecter 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 1 Processor -- start 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 1 -- start start 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 1 librados: setting wanted keys 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 1 librados: calling monclient init 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.052+0000 7f6fefe57640 10 monclient: init 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b0e80 0x7f6fe8150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe80b0e80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe80b1250 0x7f6fe80aa160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe80b1250 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe813e430 0x7f6fe80aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe813e430 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80ac630 con 0x7f6fe80b1250 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80aaa10 con 0x7f6fe813e430 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80aa720 con 0x7f6fe80b0e80 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fefe57640 10 monclient(hunting): authenticate will time out at 719.673387s 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b0e80 0x7f6fe8150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe813e430 0x7f6fe80aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b0e80 0x7f6fe8150d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.104:39370/0 (socket says 172.21.15.104:39370) 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 -- 172.21.15.104:0/1116709595 learned_addr learned my addr 172.21.15.104:0/1116709595 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): get_auth_request con 0x7f6fe80b0e80 auth_method 0 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe80b1250 0x7f6fe80aa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 10 monclient(hunting): get_auth_request con 0x7f6fe813e430 auth_method 0 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 10 monclient(hunting): get_auth_request con 0x7f6fe80b1250 auth_method 0 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient(hunting): handle_auth_done global_id 4139 payload 824 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 -- 172.21.15.104:0/1116709595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe813e430 msgr2=0x7f6fe80aae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe813e430 0x7f6fe80aae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 -- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe80b1250 msgr2=0x7f6fe80aa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe80b1250 0x7f6fe80aa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 monclient: found mon.noname-c 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fedbf3640 1 -- 172.21.15.104:0/1116709595 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4009390 con 0x7f6fe80b0e80 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fee3f4640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe813e430 0x7f6fe80aae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.053+0000 7f6fed3f2640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe80b1250 0x7f6fe80aa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fedbf3640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b0e80 0x7f6fe8150d90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400ea70 tx=0x7f6fe40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 client.?.objecter ms_handle_connect 0x7f6fe80b0e80 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:11.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 1 -- 172.21.15.104:0/1116709595 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400ca10 con 0x7f6fe80b0e80 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: dump: 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 1 -- 172.21.15.104:0/1116709595 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe400cb70 con 0x7f6fe80b0e80 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 5 monclient: authenticate success, global_id 4139 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 10 monclient: _renew_subs 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 1 -- 172.21.15.104:0/1116709595 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4013470 con 0x7f6fe80b0e80 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc009450 con 0x7f6fe80b0e80 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 monclient: dump: 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.074 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fdbfff640 4 set_mon_vals no callback set 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 10 monclient: _renew_subs 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe80a7a10 con 0x7f6fe80b0e80 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fefe57640 1 librados: init done 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 1 -- 172.21.15.104:0/1116709595 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f6fe4005970 con 0x7f6fe80b0e80 2024-09-15T02:43:11.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 1 -- 172.21.15.104:0/1116709595 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f6fe4006020 con 0x7f6fe80b0e80 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 10 client.4139.objecter ms_dispatch 0x7f6fe8000b90 osd_map(7..7 src has 1..7) v4 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 3 client.4139.objecter handle_osd_map got epochs [7,7] > 0 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 3 client.4139.objecter handle_osd_map decoding full epoch 7 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.054+0000 7f6fd9ffb640 20 client.4139.objecter dump_active .. 0 homeless 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.055+0000 7f6fefe57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.055+0000 7f6fefe57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.055+0000 7f6fefe57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.055+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe8063ff0 con 0x7f6fe80b0e80 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.057+0000 7f6fd9ffb640 1 -- 172.21.15.104:0/1116709595 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f6fe4021060 con 0x7f6fe80b0e80 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.057+0000 7f6fd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.057+0000 7f6fd9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:11.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.144+0000 7f6fefe57640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "51e17a2f-42d3-49a6-b71e-dc7e13877c2e", "id": 5}] 2024-09-15T02:43:11.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.144+0000 7f6fefe57640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "51e17a2f-42d3-49a6-b71e-dc7e13877c2e", "id": 5}] 2024-09-15T02:43:11.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.144+0000 7f6fefe57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:11.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.144+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd new", "uuid": "51e17a2f-42d3-49a6-b71e-dc7e13877c2e", "id": 5} v 0) v1 -- 0x7f6fe80a66e0 con 0x7f6fe80b0e80 2024-09-15T02:43:11.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.147+0000 7f6fd9ffb640 1 -- 172.21.15.104:0/1116709595 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "51e17a2f-42d3-49a6-b71e-dc7e13877c2e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6fe4017070 con 0x7f6fe80b0e80 2024-09-15T02:43:11.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.147+0000 7f6fd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "51e17a2f-42d3-49a6-b71e-dc7e13877c2e", "id": 5}] 2024-09-15T02:43:11.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.147+0000 7f6fd9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:11.149 INFO:teuthology.orchestra.run.smithi104.stdout:5 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 10 librados: watch_flush enter 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 10 librados: watch_flush exit 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 20 client.4139.objecter shutdown clearing up homeless session... 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 10 client.4139.objecter successfully canceled tick 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 10 monclient: shutdown 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b0e80 msgr2=0x7f6fe8150d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b0e80 0x7f6fe8150d90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400ea70 tx=0x7f6fe40090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 shutdown_connections 2024-09-15T02:43:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe813e430 0x7f6fe80aae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe80b1250 0x7f6fe80aa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 --2- 172.21.15.104:0/1116709595 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6fe80b0e80 0x7f6fe8150d90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 >> 172.21.15.104:0/1116709595 conn(0x7f6fe8062d90 msgr2=0x7f6fe80b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 shutdown_connections 2024-09-15T02:43:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 -- 172.21.15.104:0/1116709595 wait complete. 2024-09-15T02:43:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.148+0000 7f6fefe57640 1 librados: shutdown 2024-09-15T02:43:11.159 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0 6 2024-09-15T02:43:11.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.251+0000 7f99480c0640 10 monclient: get_monmap_and_config 2024-09-15T02:43:11.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.251+0000 7f99480c0640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.251+0000 7f99480c0640 10 monclient: monmap: 2024-09-15T02:43:11.252 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:11.252 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.252 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:11.251964+0000 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:11.251964+0000 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:11.253 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.251+0000 7f99480c0640 1 Processor -- start 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.251+0000 7f99480c0640 1 -- start start 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.251+0000 7f99480c0640 10 monclient: init 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99400b0e80 0x7f99400b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient(hunting): picked mon.noname-b con 0x7f99400b0e80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99400b1250 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f994013e600 0x7f99400af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient(hunting): picked mon.noname-c con 0x7f994013e600 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994013da60 con 0x7f99400b1250 2024-09-15T02:43:11.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99400b0990 con 0x7f994013e600 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400a6a30 con 0x7f99400b0e80 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f99480c0640 10 monclient(hunting): authenticate will time out at 719.872639s 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f994013e600 0x7f99400af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f994013e600 0x7f99400af3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37994/0 (socket says 172.21.15.104:37994) 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 1 -- 172.21.15.104:0/2570648495 learned_addr learned my addr 172.21.15.104:0/2570648495 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:11.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 10 monclient(hunting): get_auth_request con 0x7f994013e600 auth_method 0 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994565b640 1 --1- 172.21.15.104:0/2570648495 >> v1:172.21.15.53:6789/0 conn(0x7f99400b1250 0x7f994013def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.104:51008/0 (socket says 172.21.15.104:51008) 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 1 --2- 172.21.15.104:0/2570648495 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99400b0e80 0x7f99400b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 10 monclient(hunting): get_auth_request con 0x7f99400b0e80 auth_method 0 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f994665d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.252+0000 7f9945e5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 10 monclient(hunting): handle_auth_done global_id 4144 payload 274 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 1 -- 172.21.15.104:0/2570648495 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99400b0e80 msgr2=0x7f99400b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 1 --2- 172.21.15.104:0/2570648495 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99400b0e80 0x7f99400b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f9944e5a640 1 -- 172.21.15.104:0/2570648495 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902931574 0 0) 0x7f994013da60 con 0x7f99400b1250 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 1 -- 172.21.15.104:0/2570648495 >> v1:172.21.15.53:6789/0 conn(0x7f99400b1250 legacy=0x7f994013def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 1 monclient: found mon.noname-c 2024-09-15T02:43:11.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f994665d640 1 -- 172.21.15.104:0/2570648495 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400a5a00 con 0x7f994013e600 2024-09-15T02:43:11.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.253+0000 7f9944e5a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:11.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f994665d640 1 --2- 172.21.15.104:0/2570648495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f994013e600 0x7f99400af3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9938009f50 tx=0x7f9938009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cecff44cb223283d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 1 -- 172.21.15.104:0/2570648495 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938031400 con 0x7f994013e600 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: dump: 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.269 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 5 monclient: authenticate success, global_id 4144 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 1 -- 172.21.15.104:0/2570648495 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993802d050 con 0x7f994013e600 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 4 set_mon_vals no callback set 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 1 -- 172.21.15.104:0/2570648495 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938031790 con 0x7f994013e600 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f9944e5a640 10 monclient: dump: 2024-09-15T02:43:11.270 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 10 monclient: shutdown 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 1 -- 172.21.15.104:0/2570648495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f994013e600 msgr2=0x7f99400af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 1 --2- 172.21.15.104:0/2570648495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f994013e600 0x7f99400af3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9938009f50 tx=0x7f9938009840 comp rx=0 tx=0).stop 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 1 -- 172.21.15.104:0/2570648495 shutdown_connections 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 1 --2- 172.21.15.104:0/2570648495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f994013e600 0x7f99400af3c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 1 --2- 172.21.15.104:0/2570648495 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99400b0e80 0x7f99400b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.254+0000 7f99480c0640 1 -- 172.21.15.104:0/2570648495 >> 172.21.15.104:0/2570648495 conn(0x7f9940062d90 msgr2=0x7f9940063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.255+0000 7f99480c0640 1 -- 172.21.15.104:0/2570648495 shutdown_connections 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.255+0000 7f99480c0640 1 -- 172.21.15.104:0/2570648495 wait complete. 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.255+0000 7f99480c0640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.255+0000 7f99480c0640 10 monclient: monmap: 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:11.272 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.255+0000 7f99480c0640 1 librados: starting msgr at 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.255+0000 7f99480c0640 1 librados: starting objecter 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.255+0000 7f99480c0640 1 Processor -- start 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 -- start start 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 librados: setting wanted keys 2024-09-15T02:43:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 librados: calling monclient init 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient: init 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99400b0e80 0x7f99400a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): picked mon.noname-a con 0x7f99400b0e80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99400b1250 0x7f99400a71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): picked mon.noname-b con 0x7f99400b1250 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f994013e600 0x7f99400a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): picked mon.noname-c con 0x7f994013e600 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99400abac0 con 0x7f99400b0e80 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99400abda0 con 0x7f99400b1250 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400a8b00 con 0x7f994013e600 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f99480c0640 10 monclient(hunting): authenticate will time out at 719.876597s 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99400b1250 0x7f99400a71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99400b0e80 0x7f99400a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99400b0e80 0x7f99400a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.104:53878/0 (socket says 172.21.15.104:53878) 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99400b1250 0x7f99400a71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37996/0 (socket says 172.21.15.104:37996) 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 1 -- 172.21.15.104:0/2250211323 learned_addr learned my addr 172.21.15.104:0/2250211323 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 10 monclient(hunting): get_auth_request con 0x7f99400b0e80 auth_method 0 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): get_auth_request con 0x7f99400b1250 auth_method 0 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f994013e600 0x7f99400a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 10 monclient(hunting): get_auth_request con 0x7f994013e600 auth_method 0 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f9945e5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994665d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 -- 172.21.15.104:0/2250211323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f994013e600 msgr2=0x7f99400a76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f994013e600 0x7f99400a76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 -- 172.21.15.104:0/2250211323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99400b0e80 msgr2=0x7f99400a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99400b0e80 0x7f99400a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 monclient: found mon.noname-b 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.256+0000 7f994565b640 1 -- 172.21.15.104:0/2250211323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9938009ae0 con 0x7f99400b1250 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f994565b640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99400b1250 0x7f99400a71a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f993c000c00 tx=0x7f993c007ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 client.?.objecter ms_handle_connect 0x7f99400b1250 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 1 -- 172.21.15.104:0/2250211323 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0072e0 con 0x7f99400b1250 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: dump: 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.278 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 1 -- 172.21.15.104:0/2250211323 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c007460 con 0x7f99400b1250 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 5 monclient: authenticate success, global_id 4147 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 1 -- 172.21.15.104:0/2250211323 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0047a0 con 0x7f99400b1250 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 10 monclient: _renew_subs 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99400aa5b0 con 0x7f99400b1250 2024-09-15T02:43:11.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 monclient: dump: 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9937fff640 4 set_mon_vals no callback set 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 1 -- 172.21.15.104:0/2250211323 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f993c004bf0 con 0x7f99400b1250 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 10 monclient: _renew_subs 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99400aacf0 con 0x7f99400b1250 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f99480c0640 1 librados: init done 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 1 -- 172.21.15.104:0/2250211323 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f993c013400 con 0x7f99400b1250 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 10 client.4147.objecter ms_dispatch 0x7f9940000b90 osd_map(8..8 src has 1..8) v4 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.257+0000 7f9935ffb640 20 client.4147.objecter dump_active .. 0 homeless 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.258+0000 7f99480c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.258+0000 7f99480c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.258+0000 7f99480c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.258+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9940070fe0 con 0x7f99400b1250 2024-09-15T02:43:11.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.260+0000 7f9935ffb640 1 -- 172.21.15.104:0/2250211323 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f9940070fe0 con 0x7f99400b1250 2024-09-15T02:43:11.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.260+0000 7f9935ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.260+0000 7f9935ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:11.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.326+0000 7f99480c0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0", "id": 6}] 2024-09-15T02:43:11.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.326+0000 7f99480c0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0", "id": 6}] 2024-09-15T02:43:11.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.326+0000 7f99480c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.326+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0", "id": 6} v 0) v1 -- 0x7f99400aa7f0 con 0x7f99400b1250 2024-09-15T02:43:11.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.330+0000 7f9935ffb640 1 -- 172.21.15.104:0/2250211323 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f99400aa7f0 con 0x7f99400b1250 2024-09-15T02:43:11.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.330+0000 7f9935ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0", "id": 6}] 2024-09-15T02:43:11.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.330+0000 7f9935ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:11.331 INFO:teuthology.orchestra.run.smithi104.stdout:6 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 10 librados: watch_flush enter 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 10 librados: watch_flush exit 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 20 client.4147.objecter shutdown clearing up homeless session... 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 10 client.4147.objecter successfully canceled tick 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 10 monclient: shutdown 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99400b1250 msgr2=0x7f99400a71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99400b1250 0x7f99400a71a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f993c000c00 tx=0x7f993c007ca0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 shutdown_connections 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f994013e600 0x7f99400a76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99400b1250 0x7f99400a71a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 --2- 172.21.15.104:0/2250211323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99400b0e80 0x7f99400a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 >> 172.21.15.104:0/2250211323 conn(0x7f9940062d90 msgr2=0x7f9940142870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 shutdown_connections 2024-09-15T02:43:11.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 -- 172.21.15.104:0/2250211323 wait complete. 2024-09-15T02:43:11.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.331+0000 7f99480c0640 1 librados: shutdown 2024-09-15T02:43:11.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 7029aab4-31cc-4219-b7a4-a185564953b5 7 2024-09-15T02:43:11.426 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:11.424+0000 7f563fbc1ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T02:43:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.433+0000 7f6314cd4640 10 monclient: get_monmap_and_config 2024-09-15T02:43:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.433+0000 7f6314cd4640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.433+0000 7f6314cd4640 10 monclient: monmap: 2024-09-15T02:43:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:43:11.434227+0000 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:43:11.434227+0000 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:11.435 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.433+0000 7f6314cd4640 1 Processor -- start 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.433+0000 7f6314cd4640 1 -- start start 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.433+0000 7f6314cd4640 10 monclient: init 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63100b0e80 0x7f63100b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): picked mon.noname-a con 0x7f63100b0e80 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:11.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f631013def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): picked mon.noname-b con 0x7f63100b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f631013e430 0x7f63100af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f631013e430 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63100b0990 con 0x7f63100b0e80 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f631013e8b0 con 0x7f631013e430 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631013da60 con 0x7f63100b1250 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f6314cd4640 10 monclient(hunting): authenticate will time out at 720.054695s 2024-09-15T02:43:11.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630ed9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f631013e430 0x7f63100af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630ed9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f631013e430 0x7f63100af3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38004/0 (socket says 172.21.15.104:38004) 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630ed9d640 1 -- 172.21.15.104:0/357664963 learned_addr learned my addr 172.21.15.104:0/357664963 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630ed9d640 10 monclient(hunting): get_auth_request con 0x7f631013e430 auth_method 0 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630e59c640 1 --2- 172.21.15.104:0/357664963 >> v2:172.21.15.53:3300/0 conn(0x7f63100b0e80 0x7f63100b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630dd9b640 1 --2- 172.21.15.104:0/357664963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f631013def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630e59c640 10 monclient(hunting): get_auth_request con 0x7f63100b0e80 auth_method 0 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630dd9b640 10 monclient(hunting): get_auth_request con 0x7f63100b1250 auth_method 0 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.434+0000 7f630dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 1 -- 172.21.15.104:0/357664963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 msgr2=0x7f631013def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 1 --2- 172.21.15.104:0/357664963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f631013def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 1 -- 172.21.15.104:0/357664963 >> v2:172.21.15.53:3300/0 conn(0x7f63100b0e80 msgr2=0x7f63100b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 1 --2- 172.21.15.104:0/357664963 >> v2:172.21.15.53:3300/0 conn(0x7f63100b0e80 0x7f63100b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 1 monclient: found mon.noname-c 2024-09-15T02:43:11.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 1 -- 172.21.15.104:0/357664963 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63100a6a30 con 0x7f631013e430 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.435+0000 7f630ed9d640 1 --2- 172.21.15.104:0/357664963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f63100af3c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f63100b1a30 tx=0x7f6308009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45b55e7c75c85397 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 1 -- 172.21.15.104:0/357664963 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308031400 con 0x7f631013e430 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: dump: 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.451 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 1 -- 172.21.15.104:0/357664963 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630802d050 con 0x7f631013e430 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 5 monclient: authenticate success, global_id 4150 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 1 -- 172.21.15.104:0/357664963 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308031790 con 0x7f631013e430 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f630d59a640 10 monclient: dump: 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 4 set_mon_vals no callback set 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 10 monclient: shutdown 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 -- 172.21.15.104:0/357664963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 msgr2=0x7f63100af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 --2- 172.21.15.104:0/357664963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f63100af3c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f63100b1a30 tx=0x7f6308009840 comp rx=0 tx=0).stop 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 -- 172.21.15.104:0/357664963 shutdown_connections 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 --2- 172.21.15.104:0/357664963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f63100af3c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 --2- 172.21.15.104:0/357664963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f631013def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 --2- 172.21.15.104:0/357664963 >> v2:172.21.15.53:3300/0 conn(0x7f63100b0e80 0x7f63100b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 -- 172.21.15.104:0/357664963 >> 172.21.15.104:0/357664963 conn(0x7f6310062d90 msgr2=0x7f6310063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 -- 172.21.15.104:0/357664963 shutdown_connections 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.436+0000 7f6314cd4640 1 -- 172.21.15.104:0/357664963 wait complete. 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 10 monclient: monmap: 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:11.455 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 1 librados: starting msgr at 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 1 librados: starting objecter 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 1 Processor -- start 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 1 -- start start 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 1 librados: setting wanted keys 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 1 librados: calling monclient init 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.437+0000 7f6314cd4640 10 monclient: init 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63100b0e80 0x7f63100aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): picked mon.noname-a con 0x7f63100b0e80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f63100a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f63100b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f631013a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): picked mon.noname-b con 0x7f631013e430 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63100abf60 con 0x7f63100b0e80 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63100a7760 con 0x7f631013e430 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63100ac140 con 0x7f63100b1250 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f6314cd4640 10 monclient(hunting): authenticate will time out at 720.058341s 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f631013a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f631013a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38016/0 (socket says 172.21.15.104:38016) 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 -- 172.21.15.104:0/679146827 learned_addr learned my addr 172.21.15.104:0/679146827 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): get_auth_request con 0x7f631013e430 auth_method 0 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63100b0e80 0x7f63100aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f63100a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 10 monclient(hunting): get_auth_request con 0x7f63100b1250 auth_method 0 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 10 monclient(hunting): get_auth_request con 0x7f63100b0e80 auth_method 0 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2024-09-15T02:43:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 -- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 msgr2=0x7f63100a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f63100a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 -- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63100b0e80 msgr2=0x7f63100aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63100b0e80 0x7f63100aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 monclient: found mon.noname-b 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 -- 172.21.15.104:0/679146827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6308009ae0 con 0x7f631013e430 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f630ed9d640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f631013a1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6308009f50 tx=0x7f6308036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f62fe7fc640 10 client.?.objecter ms_handle_connect 0x7f631013e430 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f62fe7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.438+0000 7f62fe7fc640 1 -- 172.21.15.104:0/679146827 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308008770 con 0x7f631013e430 2024-09-15T02:43:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: dump: 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 1 -- 172.21.15.104:0/679146827 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63080088f0 con 0x7f631013e430 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 5 monclient: authenticate success, global_id 4153 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 10 monclient: _renew_subs 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63100aa290 con 0x7f631013e430 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 1 -- 172.21.15.104:0/679146827 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308008cb0 con 0x7f631013e430 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 monclient: dump: 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:43:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f630cd99640 4 set_mon_vals no callback set 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 10 monclient: _renew_subs 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631013a710 con 0x7f631013e430 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f6314cd4640 1 librados: init done 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 1 -- 172.21.15.104:0/679146827 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f6308038050 con 0x7f631013e430 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 1 -- 172.21.15.104:0/679146827 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f630803b620 con 0x7f631013e430 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 10 client.4153.objecter ms_dispatch 0x7f6310000b90 osd_map(9..9 src has 1..9) v4 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-09-15T02:43:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.439+0000 7f62fe7fc640 20 client.4153.objecter dump_active .. 0 homeless 2024-09-15T02:43:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.440+0000 7f6314cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.440+0000 7f6314cd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.440+0000 7f6314cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.440+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63100a78d0 con 0x7f631013e430 2024-09-15T02:43:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.441+0000 7f62fe7fc640 1 -- 172.21.15.104:0/679146827 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f6308049060 con 0x7f631013e430 2024-09-15T02:43:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.441+0000 7f62fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.441+0000 7f62fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:11.480 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:11.478+0000 7f563fbc1ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T02:43:11.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.509+0000 7f6314cd4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7029aab4-31cc-4219-b7a4-a185564953b5", "id": 7}] 2024-09-15T02:43:11.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.509+0000 7f6314cd4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7029aab4-31cc-4219-b7a4-a185564953b5", "id": 7}] 2024-09-15T02:43:11.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.509+0000 7f6314cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:11.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.509+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7029aab4-31cc-4219-b7a4-a185564953b5", "id": 7} v 0) v1 -- 0x7f6310063ff0 con 0x7f631013e430 2024-09-15T02:43:11.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.513+0000 7f62fe7fc640 1 -- 172.21.15.104:0/679146827 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7029aab4-31cc-4219-b7a4-a185564953b5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f630803f070 con 0x7f631013e430 2024-09-15T02:43:11.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.513+0000 7f62fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7029aab4-31cc-4219-b7a4-a185564953b5", "id": 7}] 2024-09-15T02:43:11.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.513+0000 7f62fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:11.515 INFO:teuthology.orchestra.run.smithi104.stdout:7 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 10 librados: watch_flush enter 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 10 librados: watch_flush exit 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 20 client.4153.objecter shutdown clearing up homeless session... 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 10 client.4153.objecter successfully canceled tick 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 10 monclient: shutdown 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 msgr2=0x7f631013a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f631013a1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6308009f50 tx=0x7f6308036000 comp rx=0 tx=0).stop 2024-09-15T02:43:11.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 shutdown_connections 2024-09-15T02:43:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f631013e430 0x7f631013a1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63100b1250 0x7f63100a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 --2- 172.21.15.104:0/679146827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63100b0e80 0x7f63100aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 >> 172.21.15.104:0/679146827 conn(0x7f6310062d90 msgr2=0x7f631013c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 shutdown_connections 2024-09-15T02:43:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 -- 172.21.15.104:0/679146827 wait complete. 2024-09-15T02:43:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-15T02:43:11.514+0000 7f6314cd4640 1 librados: shutdown 2024-09-15T02:43:11.518 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:11.515+0000 7f33631bfec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T02:43:11.525 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T02:43:11.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T02:43:11.528 INFO:tasks.ceph.osd.0:Started 2024-09-15T02:43:11.528 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T02:43:11.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T02:43:11.531 INFO:tasks.ceph.osd.1:Started 2024-09-15T02:43:11.532 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T02:43:11.532 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T02:43:11.534 INFO:tasks.ceph.osd.2:Started 2024-09-15T02:43:11.534 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T02:43:11.535 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T02:43:11.537 INFO:tasks.ceph.osd.3:Started 2024-09-15T02:43:11.537 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T02:43:11.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T02:43:11.539 INFO:tasks.ceph.osd.4:Started 2024-09-15T02:43:11.539 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T02:43:11.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T02:43:11.541 INFO:tasks.ceph.osd.5:Started 2024-09-15T02:43:11.541 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-15T02:43:11.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-15T02:43:11.543 INFO:tasks.ceph.osd.6:Started 2024-09-15T02:43:11.543 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-15T02:43:11.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-15T02:43:11.546 INFO:tasks.ceph.osd.7:Started 2024-09-15T02:43:11.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:43:11.576 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:11.575+0000 7f33631bfec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T02:43:11.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.659+0000 7ffb5703f640 10 monclient: get_monmap_and_config 2024-09-15T02:43:11.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.659+0000 7ffb5703f640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.659+0000 7ffb5703f640 10 monclient: monmap: 2024-09-15T02:43:11.661 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:11.661 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:11.660966+0000 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:11.660966+0000 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:11.662 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:11.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.659+0000 7ffb5703f640 1 Processor -- start 2024-09-15T02:43:11.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.659+0000 7ffb5703f640 1 -- start start 2024-09-15T02:43:11.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 10 monclient: init 2024-09-15T02:43:11.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb500b0fa0 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb5013b0c0 0x7ffb5013dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 10 monclient(hunting): picked mon.noname-b con 0x7ffb5013b0c0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb500a5920 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb500aed10 con 0x7ffb500b0fa0 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb54ddb640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ffb500b0fa0 0x7ffb5013a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42996/0 (socket says 172.21.15.53:42996) 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb54ddb640 1 -- 172.21.15.53:0/3558407665 learned_addr learned my addr 172.21.15.53:0/3558407665 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb4ffff640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb5013b0c0 0x7ffb5013dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.660+0000 7ffb5703f640 1 -- 172.21.15.53:0/3558407665 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb500b1370 con 0x7ffb500a5920 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb5703f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb5703f640 1 -- 172.21.15.53:0/3558407665 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb500a6b20 con 0x7ffb5013b0c0 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb5703f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb5703f640 10 monclient(hunting): authenticate will time out at 728.232101s 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): get_auth_request con 0x7ffb5013b0c0 auth_method 0 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975468821 0 0) 0x7ffb500aed10 con 0x7ffb500b0fa0 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): my global_id is 4188 2024-09-15T02:43:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb400034e0 con 0x7ffb500b0fa0 2024-09-15T02:43:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136879752 0 0) 0x7ffb500b1370 con 0x7ffb500a5920 2024-09-15T02:43:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): my global_id is 4189 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb40003800 con 0x7ffb500a5920 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172048582 0 0) 0x7ffb400034e0 con 0x7ffb500b0fa0 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb5013b0c0 msgr2=0x7ffb5013dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb5013b0c0 0x7ffb5013dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 >> v1:172.21.15.104:6789/0 conn(0x7ffb500a5920 legacy=0x7ffb5013e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.53:6789/0 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb500a6d20 con 0x7ffb500b0fa0 2024-09-15T02:43:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 65778851 0 0) 0x7ffb44001450 con 0x7ffb500b0fa0 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: dump: 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] but i'm connected to v1:172.21.15.53:6789/0, reconnecting 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 >> v1:172.21.15.53:6789/0 conn(0x7ffb500b0fa0 legacy=0x7ffb5013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb40003d60 0x7ffb40004190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): picked mon.a con 0x7ffb40003d60 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb400047d0 0x7ffb40004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): picked mon.c con 0x7ffb400047d0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb40005240 0x7ffb4000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): picked mon.b con 0x7ffb40005240 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb500a6b20 con 0x7ffb40003d60 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4000c1d0 con 0x7ffb40005240 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4000c390 con 0x7ffb400047d0 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb40003d60 0x7ffb40004190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): get_auth_request con 0x7ffb40003d60 auth_method 0 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb400047d0 0x7ffb40004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): get_auth_request con 0x7ffb400047d0 auth_method 0 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb555dc640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb40005240 0x7ffb4000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb555dc640 10 monclient(hunting): get_auth_request con 0x7ffb40005240 auth_method 0 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb555dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb555dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb555dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 10 monclient(hunting): handle_auth_done global_id 4188 payload 293 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 1 -- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb400047d0 msgr2=0x7ffb40004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb400047d0 0x7ffb40004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 1 -- 172.21.15.53:0/3558407665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb40005240 msgr2=0x7ffb4000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:11.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.661+0000 7ffb54ddb640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb40005240 0x7ffb4000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 1 monclient: found mon.a 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 1 -- 172.21.15.53:0/3558407665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb500a6d20 con 0x7ffb40003d60 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb5703f640 5 monclient: authenticate success, global_id 4188 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb5703f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb54ddb640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb40003d60 0x7ffb40004190 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb44013a60 tx=0x7ffb440131e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42f6453b60c703d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44012030 con 0x7ffb40003d60 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb4f7fe640 10 monclient: dump: 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb44033c40 con 0x7ffb40003d60 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.662+0000 7ffb4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb4f7fe640 1 -- 172.21.15.53:0/3558407665 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4403b440 con 0x7ffb40003d60 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb4f7fe640 10 monclient: dump: 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 4 set_mon_vals no callback set 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 10 monclient: shutdown 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 1 -- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb40003d60 msgr2=0x7ffb40004190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 1 --2- 172.21.15.53:0/3558407665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb40003d60 0x7ffb40004190 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb44013a60 tx=0x7ffb440131e0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb555dc640 1 -- 172.21.15.53:0/3558407665 reap_dead start 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 1 -- 172.21.15.53:0/3558407665 shutdown_connections 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 1 -- 172.21.15.53:0/3558407665 >> 172.21.15.53:0/3558407665 conn(0x7ffb50062d90 msgr2=0x7ffb50063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 1 -- 172.21.15.53:0/3558407665 shutdown_connections 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.663+0000 7ffb5703f640 1 -- 172.21.15.53:0/3558407665 wait complete. 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.664+0000 7ffb5703f640 10 monclient: build_initial_monmap 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.664+0000 7ffb5703f640 10 monclient: monmap: 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:11.687 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.664+0000 7ffb5703f640 1 librados: starting msgr at 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.664+0000 7ffb5703f640 1 librados: starting objecter 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.664+0000 7ffb5703f640 1 Processor -- start 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 -- start start 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 librados: setting wanted keys 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 librados: calling monclient init 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient: init 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb40003d60 0x7ffb500b4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): picked mon.noname-c con 0x7ffb40003d60 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb5014c2e0 0x7ffb5006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): picked mon.noname-b con 0x7ffb5014c2e0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb5006cb40 0x7ffb5006cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): picked mon.noname-a con 0x7ffb5006cb40 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5006c5b0 con 0x7ffb5006cb40 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb54ddb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb40003d60 0x7ffb500b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5006c300 con 0x7ffb5014c2e0 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5006ded0 con 0x7ffb40003d60 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb555dc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb5006cb40 0x7ffb5006cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.665+0000 7ffb5703f640 10 monclient(hunting): authenticate will time out at 728.236993s 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb40003d60 0x7ffb500b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:48702/0 (socket says 172.21.15.53:48702) 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 1 -- 172.21.15.53:0/2033205091 learned_addr learned my addr 172.21.15.53:0/2033205091 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:11.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb5014c2e0 0x7ffb5006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:11.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): get_auth_request con 0x7ffb5006cb40 auth_method 0 2024-09-15T02:43:11.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 10 monclient(hunting): get_auth_request con 0x7ffb5014c2e0 auth_method 0 2024-09-15T02:43:11.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 10 monclient(hunting): get_auth_request con 0x7ffb40003d60 auth_method 0 2024-09-15T02:43:11.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:11.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:11.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:11.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:11.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:11.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:11.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient(hunting): handle_auth_done global_id 4191 payload 824 2024-09-15T02:43:11.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 1 -- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb40003d60 msgr2=0x7ffb500b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb40003d60 0x7ffb500b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 1 -- 172.21.15.53:0/2033205091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb5014c2e0 msgr2=0x7ffb5006bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb5014c2e0 0x7ffb5006bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient: _finish_hunting 0 2024-09-15T02:43:11.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 1 monclient: found mon.noname-a 2024-09-15T02:43:11.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:11.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:11.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 1 -- 172.21.15.53:0/2033205091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5006e1b0 con 0x7ffb5006cb40 2024-09-15T02:43:11.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4ffff640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb5014c2e0 0x7ffb5006bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:11.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb54ddb640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb40003d60 0x7ffb500b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:11.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb555dc640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb5006cb40 0x7ffb5006cfc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffb500690c0 tx=0x7ffb48009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:11.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4cff9640 10 client.?.objecter ms_handle_connect 0x7ffb5006cb40 2024-09-15T02:43:11.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.666+0000 7ffb4cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:11.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 1 -- 172.21.15.53:0/2033205091 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4800edd0 con 0x7ffb5006cb40 2024-09-15T02:43:11.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:11.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: dump: 2024-09-15T02:43:11.699 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:11.699 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.699 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.699 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.699 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.699 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:11.700 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.700 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.700 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.700 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:11.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: _finish_auth 0 2024-09-15T02:43:11.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:11.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 1 -- 172.21.15.53:0/2033205091 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb48006b10 con 0x7ffb5006cb40 2024-09-15T02:43:11.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 5 monclient: authenticate success, global_id 4191 2024-09-15T02:43:11.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 10 monclient: _renew_subs 2024-09-15T02:43:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5006a400 con 0x7ffb5006cb40 2024-09-15T02:43:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 1 -- 172.21.15.53:0/2033205091 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb480030d0 con 0x7ffb5006cb40 2024-09-15T02:43:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 monclient: dump: 2024-09-15T02:43:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:11.704 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:11.704 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:11.704 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:11.704 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:11.704 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:11.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4effd640 4 set_mon_vals no callback set 2024-09-15T02:43:11.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:11.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 10 monclient: _renew_subs 2024-09-15T02:43:11.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:11.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5006a850 con 0x7ffb5006cb40 2024-09-15T02:43:11.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb5703f640 1 librados: init done 2024-09-15T02:43:11.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 1 -- 172.21.15.53:0/2033205091 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ffb480016e0 con 0x7ffb5006cb40 2024-09-15T02:43:11.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 1 -- 172.21.15.53:0/2033205091 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7ffb480034c0 con 0x7ffb5006cb40 2024-09-15T02:43:11.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 10 client.4191.objecter ms_dispatch 0x7ffb50000b90 osd_map(10..10 src has 1..10) v4 2024-09-15T02:43:11.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-09-15T02:43:11.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-09-15T02:43:11.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.667+0000 7ffb4cff9640 20 client.4191.objecter dump_active .. 0 homeless 2024-09-15T02:43:11.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.668+0000 7ffb5703f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.668+0000 7ffb5703f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.668+0000 7ffb5703f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:11.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.668+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb500646e0 con 0x7ffb5006cb40 2024-09-15T02:43:11.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.670+0000 7ffb4cff9640 1 -- 172.21.15.53:0/2033205091 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7ffb4801d060 con 0x7ffb5006cb40 2024-09-15T02:43:11.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.670+0000 7ffb4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:11.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.670+0000 7ffb4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:11.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.727+0000 7ffb5703f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:11.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.727+0000 7ffb5703f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:11.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.727+0000 7ffb5703f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:11.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.727+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb5006eff0 con 0x7ffb5006cb40 2024-09-15T02:43:11.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.728+0000 7ffb4cff9640 1 -- 172.21.15.53:0/2033205091 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7ffb48027090 con 0x7ffb5006cb40 2024-09-15T02:43:11.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.728+0000 7ffb4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:11.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.728+0000 7ffb4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:11.731 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:43:11.731 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":10,"fsid":"7e7cb607-2b58-4a36-a082-99425792b70f","created":"2024-09-15T02:43:09.455959+0000","modified":"2024-09-15T02:43:11.511047+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T02:43:11.511047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ab035ba6-bb97-46de-9069-4082e1295127","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb614e79-5e04-45a4-9962-84c7619ddb87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"08ea49a0-28e1-4740-a4b4-d722560b15c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1a3a89c3-b449-46ba-bc65-3f53371d3af1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d969fe78-c8c8-4c03-a534-2173406b0ba6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51e17a2f-42d3-49a6-b71e-dc7e13877c2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7029aab4-31cc-4219-b7a4-a185564953b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.728+0000 7ffb5703f640 10 librados: watch_flush enter 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.728+0000 7ffb5703f640 10 librados: watch_flush exit 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 20 client.4191.objecter shutdown clearing up homeless session... 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 10 client.4191.objecter successfully canceled tick 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 10 monclient: shutdown 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb5006cb40 msgr2=0x7ffb5006cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb5006cb40 0x7ffb5006cfc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffb500690c0 tx=0x7ffb48009040 comp rx=0 tx=0).stop 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 shutdown_connections 2024-09-15T02:43:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb5006cb40 0x7ffb5006cfc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb5014c2e0 0x7ffb5006bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 --2- 172.21.15.53:0/2033205091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb40003d60 0x7ffb500b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:11.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 >> 172.21.15.53:0/2033205091 conn(0x7ffb50062d90 msgr2=0x7ffb50139e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:11.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 shutdown_connections 2024-09-15T02:43:11.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 -- 172.21.15.53:0/2033205091 wait complete. 2024-09-15T02:43:11.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:11.729+0000 7ffb5703f640 1 librados: shutdown 2024-09-15T02:43:11.741 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T02:43:11.742 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T02:43:11.896 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:11.893+0000 7f563fbc1ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T02:43:11.957 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:11.955+0000 7f33631bfec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T02:43:11.981 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:11.979+0000 7f563fbc1ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T02:43:12.038 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.037+0000 7f33631bfec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T02:43:12.073 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.071+0000 7f563fbc1ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:43:12.121 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.118+0000 7f563fbc1ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T02:43:12.126 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.125+0000 7f33631bfec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:43:12.141 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-09-15T02:43:12.140+0000 7f06acb2b440 -1 Falling back to public interface 2024-09-15T02:43:12.144 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-09-15T02:43:12.143+0000 7f527e926440 -1 Falling back to public interface 2024-09-15T02:43:12.155 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-09-15T02:43:12.152+0000 7f337d209440 -1 Falling back to public interface 2024-09-15T02:43:12.155 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:43:12.152+0000 7f9a1dd88440 -1 Falling back to public interface 2024-09-15T02:43:12.156 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-09-15T02:43:12.152+0000 7fa354d21440 -1 Falling back to public interface 2024-09-15T02:43:12.156 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-09-15T02:43:12.153+0000 7f337b1fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-09-15T02:43:12.156 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-09-15T02:43:12.153+0000 7fa352ffd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-09-15T02:43:12.157 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:43:12.153+0000 7f9a1bffd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6806/0: (98) Address already in use 2024-09-15T02:43:12.157 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:43:12.153+0000 7f9a1bffd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6807/0: (98) Address already in use 2024-09-15T02:43:12.158 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-09-15T02:43:12.156+0000 7f00c511c440 -1 Falling back to public interface 2024-09-15T02:43:12.160 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:43:12.158+0000 7f227f80e440 -1 Falling back to public interface 2024-09-15T02:43:12.162 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-09-15T02:43:12.161+0000 7fc9cfc13440 -1 Falling back to public interface 2024-09-15T02:43:12.175 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.174+0000 7f33631bfec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T02:43:12.222 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.219+0000 7f563fbc1ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T02:43:12.270 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.268+0000 7f33631bfec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T02:43:12.276 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.274+0000 7f563fbc1ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T02:43:12.321 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.320+0000 7f33631bfec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T02:43:12.324 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.320+0000 7f563fbc1ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T02:43:12.364 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.363+0000 7f33631bfec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T02:43:12.419 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.417+0000 7f563fbc1ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T02:43:12.459 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.458+0000 7f33631bfec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T02:43:12.471 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.469+0000 7f563fbc1ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T02:43:12.508 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.507+0000 7f33631bfec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T02:43:12.524 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.522+0000 7f563fbc1ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T02:43:12.551 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T02:43:12.558 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.557+0000 7f33631bfec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T02:43:12.595 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.593+0000 7f563fbc1ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T02:43:12.623 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.622+0000 7f33631bfec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.637+0000 7fa88f554640 10 monclient: get_monmap_and_config 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.637+0000 7fa88f554640 10 monclient: build_initial_monmap 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.637+0000 7fa88f554640 10 monclient: monmap: 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:epoch 0 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:43:12.638820+0000 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:43:12.638820+0000 2024-09-15T02:43:12.639 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:12.640 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:12.640 INFO:teuthology.misc.health.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:12.640 INFO:teuthology.misc.health.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:12.640 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:12.640 INFO:teuthology.misc.health.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:12.640 INFO:teuthology.misc.health.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:12.640 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:12.643 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.637+0000 7fa88f554640 1 Processor -- start 2024-09-15T02:43:12.643 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.637+0000 7fa88f554640 1 -- start start 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.637+0000 7fa88f554640 10 monclient: init 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa8880b1290 0x7fa8880b0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient(hunting): picked mon.noname-c con 0x7fa8880b1290 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8880b0740 0x7fa8880af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient(hunting): picked mon.noname-a con 0x7fa8880b0740 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8880afc10 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa888140e40 con 0x7fa8880afc10 2024-09-15T02:43:12.644 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8880b0070 con 0x7fa8880b0740 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8880a6e60 con 0x7fa8880b1290 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88f554640 10 monclient(hunting): authenticate will time out at 729.209512s 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8880b0740 0x7fa8880af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8880b0740 0x7fa8880af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39750/0 (socket says 172.21.15.53:39750) 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 1 -- 172.21.15.53:0/1190632731 learned_addr learned my addr 172.21.15.53:0/1190632731 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): get_auth_request con 0x7fa8880b0740 auth_method 0 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:12.645 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 1 --2- 172.21.15.53:0/1190632731 >> v2:172.21.15.104:3300/0 conn(0x7fa8880b1290 0x7fa8880b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 10 monclient(hunting): get_auth_request con 0x7fa8880b1290 auth_method 0 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:12.646 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88d2f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.638+0000 7fa88caef640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 1 -- 172.21.15.53:0/1190632731 >> v2:172.21.15.104:3300/0 conn(0x7fa8880b1290 msgr2=0x7fa8880b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 1 --2- 172.21.15.53:0/1190632731 >> v2:172.21.15.104:3300/0 conn(0x7fa8880b1290 0x7fa8880b0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 1 -- 172.21.15.53:0/1190632731 >> v1:172.21.15.104:6789/0 conn(0x7fa8880afc10 legacy=0x7fa8881412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 10 monclient: _finish_hunting 0 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 1 monclient: found mon.noname-a 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 1 -- 172.21.15.53:0/1190632731 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8880b1660 con 0x7fa8880b0740 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88caef640 1 --2- 172.21.15.53:0/1190632731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880af6d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8880697e0 tx=0x7fa8840091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2945a6ea128cab46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 1 -- 172.21.15.53:0/1190632731 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88402ae80 con 0x7fa8880b0740 2024-09-15T02:43:12.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: dump: 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:12.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: _finish_auth 0 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 1 -- 172.21.15.53:0/1190632731 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa884032440 con 0x7fa8880b0740 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88f554640 5 monclient: authenticate success, global_id 4194 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88f554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 1 -- 172.21.15.53:0/1190632731 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884032800 con 0x7fa8880b0740 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88f554640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa88f554640 4 set_mon_vals no callback set 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.639+0000 7fa87bfff640 10 monclient: dump: 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:12.649 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.650 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.650 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:12.650 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:12.650 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:12.650 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:12.650 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:12.650 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:12.650 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.640+0000 7f563fbc1ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 10 monclient: shutdown 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 -- 172.21.15.53:0/1190632731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 msgr2=0x7fa8880af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 --2- 172.21.15.53:0/1190632731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880af6d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8880697e0 tx=0x7fa8840091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 -- 172.21.15.53:0/1190632731 shutdown_connections 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 --2- 172.21.15.53:0/1190632731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880af6d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 --2- 172.21.15.53:0/1190632731 >> v2:172.21.15.104:3300/0 conn(0x7fa8880b1290 0x7fa8880b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 -- 172.21.15.53:0/1190632731 >> 172.21.15.53:0/1190632731 conn(0x7fa888062e60 msgr2=0x7fa888063260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:12.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 -- 172.21.15.53:0/1190632731 shutdown_connections 2024-09-15T02:43:12.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.640+0000 7fa88f554640 1 -- 172.21.15.53:0/1190632731 wait complete. 2024-09-15T02:43:12.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 10 monclient: build_initial_monmap 2024-09-15T02:43:12.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 10 monclient: monmap: 2024-09-15T02:43:12.656 INFO:teuthology.misc.health.smithi053.stderr:epoch 0 2024-09-15T02:43:12.656 INFO:teuthology.misc.health.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:12.656 INFO:teuthology.misc.health.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:created 0.000000 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 1 librados: starting msgr at 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 1 librados: starting objecter 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 1 Processor -- start 2024-09-15T02:43:12.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 1 -- start start 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 1 librados: setting wanted keys 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 1 librados: calling monclient init 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.641+0000 7fa88f554640 10 monclient: init 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8880afc10 0x7fa888151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): picked mon.noname-b con 0x7fa8880afc10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): picked mon.noname-a con 0x7fa8880b0740 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880b1290 0x7fa8880aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): picked mon.noname-c con 0x7fa8880b1290 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880aa240 con 0x7fa8880b0740 2024-09-15T02:43:12.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880aa4f0 con 0x7fa8880afc10 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880ac160 con 0x7fa8880b1290 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88f554640 10 monclient(hunting): authenticate will time out at 729.213399s 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880a9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39760/0 (socket says 172.21.15.53:39760) 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 -- 172.21.15.53:0/2584582958 learned_addr learned my addr 172.21.15.53:0/2584582958 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): get_auth_request con 0x7fa8880b0740 auth_method 0 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880b1290 0x7fa8880aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:12.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8880afc10 0x7fa888151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 10 monclient(hunting): get_auth_request con 0x7fa8880b1290 auth_method 0 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 10 monclient(hunting): get_auth_request con 0x7fa8880afc10 auth_method 0 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:12.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88daf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 -- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880b1290 msgr2=0x7fa8880aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880b1290 0x7fa8880aa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 -- 172.21.15.53:0/2584582958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8880afc10 msgr2=0x7fa888151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8880afc10 0x7fa888151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient: _finish_hunting 0 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 monclient: found mon.noname-a 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:12.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88caef640 1 -- 172.21.15.53:0/2584582958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa884009450 con 0x7fa8880b0740 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.642+0000 7fa88d2f0640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8880afc10 0x7fa888151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88caef640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880a9d00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa884009940 tx=0x7fa884032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 client.?.objecter ms_handle_connect 0x7fa8880b0740 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 1 -- 172.21.15.53:0/2584582958 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884031bb0 con 0x7fa8880b0740 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: dump: 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:12.662 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: _finish_auth 0 2024-09-15T02:43:12.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 1 -- 172.21.15.53:0/2584582958 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa884031d30 con 0x7fa8880b0740 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 5 monclient: authenticate success, global_id 4197 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 10 monclient: _renew_subs 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa888151ca0 con 0x7fa8880b0740 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 1 -- 172.21.15.53:0/2584582958 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884031ea0 con 0x7fa8880b0740 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 monclient: dump: 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:12.664 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa87b7fe640 4 set_mon_vals no callback set 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 10 monclient: _renew_subs 2024-09-15T02:43:12.665 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8880ab0e0 con 0x7fa8880b0740 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa88f554640 1 librados: init done 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 1 -- 172.21.15.53:0/2584582958 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fa884039040 con 0x7fa8880b0740 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 1 -- 172.21.15.53:0/2584582958 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fa88403dda0 con 0x7fa8880b0740 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 10 client.4197.objecter ms_dispatch 0x7fa888000b90 osd_map(10..10 src has 1..10) v4 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.643+0000 7fa8797fa640 20 client.4197.objecter dump_active .. 0 homeless 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.644+0000 7fa88f554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.644+0000 7fa88f554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.644+0000 7fa88f554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.644+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa888141cc0 con 0x7fa8880b0740 2024-09-15T02:43:12.666 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.646+0000 7fa8797fa640 1 -- 172.21.15.53:0/2584582958 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fa884042070 con 0x7fa8880b0740 2024-09-15T02:43:12.667 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.646+0000 7fa8797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:12.667 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.646+0000 7fa8797fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:12.667 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.665+0000 7f33631bfec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T02:43:12.701 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:12.699+0000 7f563fbc1ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T02:43:12.709 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:12.708+0000 7f33631bfec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T02:43:12.725 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.723+0000 7fa88f554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:12.725 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.723+0000 7fa88f554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:12.726 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.723+0000 7fa88f554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:12.726 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.723+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8880b0e40 con 0x7fa8880b0740 2024-09-15T02:43:12.726 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.723+0000 7fa8797fa640 1 -- 172.21.15.53:0/2584582958 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fa884037070 con 0x7fa8880b0740 2024-09-15T02:43:12.726 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.723+0000 7fa8797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:12.726 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.723+0000 7fa8797fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:12.726 INFO:teuthology.misc.health.smithi053.stdout: 2024-09-15T02:43:12.726 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":10,"fsid":"7e7cb607-2b58-4a36-a082-99425792b70f","created":"2024-09-15T02:43:09.455959+0000","modified":"2024-09-15T02:43:11.511047+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T02:43:11.511047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ab035ba6-bb97-46de-9069-4082e1295127","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb614e79-5e04-45a4-9962-84c7619ddb87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"08ea49a0-28e1-4740-a4b4-d722560b15c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1a3a89c3-b449-46ba-bc65-3f53371d3af1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d969fe78-c8c8-4c03-a534-2173406b0ba6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51e17a2f-42d3-49a6-b71e-dc7e13877c2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7029aab4-31cc-4219-b7a4-a185564953b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 10 librados: watch_flush enter 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 10 librados: watch_flush exit 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 20 client.4197.objecter shutdown clearing up homeless session... 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 10 client.4197.objecter successfully canceled tick 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 10 monclient: shutdown 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 msgr2=0x7fa8880a9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880a9d00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa884009940 tx=0x7fa884032000 comp rx=0 tx=0).stop 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 shutdown_connections 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880b1290 0x7fa8880aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.728 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880b0740 0x7fa8880a9d00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.729 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 1 --2- 172.21.15.53:0/2584582958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8880afc10 0x7fa888151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:12.729 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.724+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 >> 172.21.15.53:0/2584582958 conn(0x7fa888062e60 msgr2=0x7fa88813faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:12.729 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.725+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 shutdown_connections 2024-09-15T02:43:12.729 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.725+0000 7fa88f554640 1 -- 172.21.15.53:0/2584582958 wait complete. 2024-09-15T02:43:12.729 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:12.725+0000 7fa88f554640 1 librados: shutdown 2024-09-15T02:43:12.735 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-15T02:43:13.322 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:13.321+0000 7f33631bfec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T02:43:13.353 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-09-15T02:43:13.352+0000 7f563fbc1ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T02:43:14.338 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:14.336+0000 7f334770d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T02:43:14.338 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-09-15T02:43:14.337+0000 7f334770d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T02:43:15.612 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-09-15T02:43:15.608+0000 7f337d209440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:43:15.638 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-09-15T02:43:15.637+0000 7f06acb2b440 -1 osd.5 0 log_to_monitors true 2024-09-15T02:43:15.673 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-09-15T02:43:15.672+0000 7fa354d21440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:43:15.674 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-09-15T02:43:15.672+0000 7f00c511c440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:43:15.685 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-09-15T02:43:15.684+0000 7f527e926440 -1 osd.7 0 log_to_monitors true 2024-09-15T02:43:15.686 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:43:15.685+0000 7f9a1dd88440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:43:15.702 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:43:15.701+0000 7f227f80e440 -1 osd.6 0 log_to_monitors true 2024-09-15T02:43:15.708 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-09-15T02:43:15.707+0000 7fc9cfc13440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:43:17.352 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-09-15T02:43:17.350+0000 7f00bfbf6640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T02:43:17.352 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-09-15T02:43:17.350+0000 7f527abf9640 -1 osd.7 0 waiting for initial osdmap 2024-09-15T02:43:17.352 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-09-15T02:43:17.350+0000 7f33791f9640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T02:43:17.352 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-09-15T02:43:17.351+0000 7fa34f7f6640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T02:43:17.354 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-09-15T02:43:17.353+0000 7f06a75f6640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T02:43:17.354 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:43:17.353+0000 7f2279ff6640 -1 osd.6 0 waiting for initial osdmap 2024-09-15T02:43:17.354 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-09-15T02:43:17.353+0000 7fc9ca3f6640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T02:43:17.356 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-09-15T02:43:17.355+0000 7f5273a87640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:17.356 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:43:17.355+0000 7f9a187f6640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T02:43:17.356 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-09-15T02:43:17.355+0000 7f33721eb640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:17.357 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-09-15T02:43:17.355+0000 7f00ba287640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:17.357 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-09-15T02:43:17.356+0000 7fa349feb640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:17.358 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-09-15T02:43:17.357+0000 7f06a1deb640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:17.358 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-09-15T02:43:17.357+0000 7fc9c4beb640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:17.359 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:43:17.358+0000 7f22747eb640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:17.360 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:43:17.359+0000 7f9a12feb640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:43:19.542 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T02:43:19.638 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.637+0000 7fb14375d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:19.638 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.637+0000 7fb14375d640 10 monclient: build_initial_monmap 2024-09-15T02:43:19.638 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.637+0000 7fb14375d640 10 monclient: monmap: 2024-09-15T02:43:19.638 INFO:teuthology.misc.health.smithi053.stderr:epoch 0 2024-09-15T02:43:19.638 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:43:19.638121+0000 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:43:19.638121+0000 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:19.639 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:19.647 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.637+0000 7fb14375d640 1 Processor -- start 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 1 -- start start 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient: init 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c06f0d0 0x7fb13c0ab290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient(hunting): picked mon.noname-b con 0x7fb13c06f0d0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb13c06f4a0 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb13c067ce0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c0aa900 con 0x7fb13c06f4a0 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c06a200 con 0x7fb13c067ce0 2024-09-15T02:43:19.648 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0aad20 con 0x7fb13c06f0d0 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.638+0000 7fb14375d640 10 monclient(hunting): authenticate will time out at 736.208919s 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c06f0d0 0x7fb13c0ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb140cf8640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb13c06f4a0 0x7fb13c0675d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40658/0 (socket says 172.21.15.53:40658) 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb140cf8640 1 -- 172.21.15.53:0/2965273413 learned_addr learned my addr 172.21.15.53:0/2965273413 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): get_auth_request con 0x7fb13c06f0d0 auth_method 0 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.649 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb13bfff640 1 -- 172.21.15.53:0/2965273413 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110593655 0 0) 0x7fb13c0aa900 con 0x7fb13c06f4a0 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb13bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb13bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb13bfff640 10 monclient(hunting): my global_id is 4242 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb13bfff640 1 -- 172.21.15.53:0/2965273413 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1240034e0 con 0x7fb13c06f4a0 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 1 -- 172.21.15.53:0/2965273413 >> v1:172.21.15.104:6789/0 conn(0x7fb13c067ce0 legacy=0x7fb13c1481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 1 -- 172.21.15.53:0/2965273413 >> v1:172.21.15.53:6789/0 conn(0x7fb13c06f4a0 legacy=0x7fb13c0675d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient: _finish_hunting 0 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 1 monclient: found mon.noname-b 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:19.650 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.639+0000 7fb1414f9640 1 -- 172.21.15.53:0/2965273413 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c13b3d0 con 0x7fb13c06f0d0 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb1414f9640 1 --2- 172.21.15.53:0/2965273413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c06f0d0 0x7fb13c0ab290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009880 tx=0x7fb12c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f6dd820ab8fc9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 1 -- 172.21.15.53:0/2965273413 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c02ace0 con 0x7fb13c06f0d0 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: dump: 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.651 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: _finish_auth 0 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 1 -- 172.21.15.53:0/2965273413 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c02ae40 con 0x7fb13c06f0d0 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 5 monclient: authenticate success, global_id 4217 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 1 -- 172.21.15.53:0/2965273413 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c032670 con 0x7fb13c06f0d0 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 4 set_mon_vals no callback set 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:19.652 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb13bfff640 10 monclient: dump: 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 10 monclient: shutdown 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 1 -- 172.21.15.53:0/2965273413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c06f0d0 msgr2=0x7fb13c0ab290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.653 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.640+0000 7fb14375d640 1 --2- 172.21.15.53:0/2965273413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c06f0d0 0x7fb13c0ab290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009880 tx=0x7fb12c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 1 -- 172.21.15.53:0/2965273413 shutdown_connections 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 1 --2- 172.21.15.53:0/2965273413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c06f0d0 0x7fb13c0ab290 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 1 -- 172.21.15.53:0/2965273413 >> 172.21.15.53:0/2965273413 conn(0x7fb13c0b1fe0 msgr2=0x7fb13c0ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 1 -- 172.21.15.53:0/2965273413 shutdown_connections 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 1 -- 172.21.15.53:0/2965273413 wait complete. 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 10 monclient: build_initial_monmap 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 10 monclient: monmap: 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:epoch 0 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:created 0.000000 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:19.654 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 1 librados: starting msgr at 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.641+0000 7fb14375d640 1 librados: starting objecter 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 Processor -- start 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 -- start start 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 librados: setting wanted keys 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 librados: calling monclient init 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient: init 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:19.655 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c067ce0 0x7fb13c13d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): picked mon.noname-c con 0x7fb13c067ce0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb13c06f0d0 0x7fb13c139ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): picked mon.noname-a con 0x7fb13c06f0d0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13c06f4a0 0x7fb13c13a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): picked mon.noname-b con 0x7fb13c06f4a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c13f260 con 0x7fb13c06f0d0 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c1399c0 con 0x7fb13c06f4a0 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c13bcc0 con 0x7fb13c067ce0 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:19.656 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb14375d640 10 monclient(hunting): authenticate will time out at 736.212819s 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb1414f9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c067ce0 0x7fb13c13d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb140cf8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb13c06f0d0 0x7fb13c139ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb1414f9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c067ce0 0x7fb13c13d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:46922/0 (socket says 172.21.15.53:46922) 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb1414f9640 1 -- 172.21.15.53:0/2899596499 learned_addr learned my addr 172.21.15.53:0/2899596499 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb1414f9640 10 monclient(hunting): get_auth_request con 0x7fb13c067ce0 auth_method 0 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb1414f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb1414f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb1414f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb140cf8640 10 monclient(hunting): get_auth_request con 0x7fb13c06f0d0 auth_method 0 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb140cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb140cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.642+0000 7fb140cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.657 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13c06f4a0 0x7fb13c13a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 10 monclient(hunting): get_auth_request con 0x7fb13c06f4a0 auth_method 0 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1414f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1414f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1414f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.658 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb141cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 1 -- 172.21.15.53:0/2899596499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c067ce0 msgr2=0x7fb13c13d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c067ce0 0x7fb13c13d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 1 -- 172.21.15.53:0/2899596499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13c06f4a0 msgr2=0x7fb13c13a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13c06f4a0 0x7fb13c13a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 10 monclient: _finish_hunting 0 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 1 monclient: found mon.noname-a 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 1 -- 172.21.15.53:0/2899596499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c009390 con 0x7fb13c06f0d0 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb140cf8640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb13c06f0d0 0x7fb13c139ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb13000cc20 tx=0x7fb130001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 client.?.objecter ms_handle_connect 0x7fb13c06f0d0 2024-09-15T02:43:19.659 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 1 -- 172.21.15.53:0/2899596499 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000ce40 con 0x7fb13c06f0d0 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: dump: 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.660 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: _finish_auth 0 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 1 -- 172.21.15.53:0/2899596499 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13000a850 con 0x7fb13c06f0d0 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 5 monclient: authenticate success, global_id 4245 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 1 -- 172.21.15.53:0/2899596499 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000ac50 con 0x7fb13c06f0d0 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 10 monclient: _renew_subs 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.661 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c1395e0 con 0x7fb13c06f0d0 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb13b7fe640 4 set_mon_vals no callback set 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb1397fa640 10 monclient: dump: 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:epoch 1 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.662 INFO:teuthology.misc.health.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr: 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 10 monclient: _renew_subs 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c13ab00 con 0x7fb13c06f0d0 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.643+0000 7fb14375d640 1 librados: init done 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.644+0000 7fb14375d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.644+0000 7fb14375d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.644+0000 7fb14375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.644+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c139c00 con 0x7fb13c06f0d0 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1397fa640 1 -- 172.21.15.53:0/2899596499 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 89555+0+0 (secure 0 0 0) 0x7fb130007300 con 0x7fb13c06f0d0 2024-09-15T02:43:19.663 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1397fa640 1 --2- 172.21.15.53:0/2899596499 >> v2:172.21.15.104:6816/32804 conn(0x7fb12806e7f0 0x7fb128070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1397fa640 1 -- 172.21.15.53:0/2899596499 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3722+0+0 (secure 0 0 0) 0x7fb13008b490 con 0x7fb13c06f0d0 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1397fa640 10 client.4245.objecter ms_dispatch 0x7fb13c000b90 osd_map(14..14 src has 1..14) v4 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1397fa640 3 client.4245.objecter handle_osd_map got epochs [14,14] > 0 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1397fa640 3 client.4245.objecter handle_osd_map decoding full epoch 14 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1397fa640 20 client.4245.objecter dump_active .. 0 homeless 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.645+0000 7fb1414f9640 1 --2- 172.21.15.53:0/2899596499 >> v2:172.21.15.104:6816/32804 conn(0x7fb12806e7f0 0x7fb128070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.646+0000 7fb1414f9640 10 monclient: get_auth_request con 0x7fb12806e7f0 auth_method 0 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.646+0000 7fb1414f9640 1 --2- 172.21.15.53:0/2899596499 >> v2:172.21.15.104:6816/32804 conn(0x7fb12806e7f0 0x7fb128070ca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb12c035040 tx=0x7fb12c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:19.664 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.646+0000 7fb1397fa640 10 client.4245.objecter ms_handle_connect 0x7fb12806e7f0 2024-09-15T02:43:19.665 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.650+0000 7fb1397fa640 1 -- 172.21.15.53:0/2899596499 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb13c139c00 con 0x7fb13c06f0d0 2024-09-15T02:43:19.665 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.650+0000 7fb1397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:19.665 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.650+0000 7fb1397fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:19.816 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.815+0000 7fb14375d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:19.816 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.815+0000 7fb14375d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:19.816 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.815+0000 7fb14375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.816 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.815+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb13c139c00 con 0x7fb13c06f0d0 2024-09-15T02:43:19.817 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.815+0000 7fb1397fa640 1 -- 172.21.15.53:0/2899596499 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10355 (secure 0 0 0) 0x7fb13c139c00 con 0x7fb13c06f0d0 2024-09-15T02:43:19.817 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.815+0000 7fb1397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:19.817 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.815+0000 7fb1397fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:19.817 INFO:teuthology.misc.health.smithi053.stdout: 2024-09-15T02:43:19.817 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":14,"fsid":"7e7cb607-2b58-4a36-a082-99425792b70f","created":"2024-09-15T02:43:09.455959+0000","modified":"2024-09-15T02:43:19.374225+0000","last_up_change":"2024-09-15T02:43:18.346896+0000","last_in_change":"2024-09-15T02:43:11.511047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:43:19.373532+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"ab035ba6-bb97-46de-9069-4082e1295127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":33156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":33156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":33156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":33156}]},"public_addr":"172.21.15.53:6801/33156","cluster_addr":"172.21.15.53:6804/33156","heartbeat_back_addr":"172.21.15.53:6809/33156","heartbeat_front_addr":"172.21.15.53:6806/33156","state":["exists","up"]},{"osd":1,"uuid":"eb614e79-5e04-45a4-9962-84c7619ddb87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":33157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":33157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":33157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":33157}]},"public_addr":"172.21.15.53:6800/33157","cluster_addr":"172.21.15.53:6803/33157","heartbeat_back_addr":"172.21.15.53:6811/33157","heartbeat_front_addr":"172.21.15.53:6808/33157","state":["exists","up"]},{"osd":2,"uuid":"08ea49a0-28e1-4740-a4b4-d722560b15c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":33158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":33158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":33158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":33158}]},"public_addr":"172.21.15.53:6802/33158","cluster_addr":"172.21.15.53:6805/33158","heartbeat_back_addr":"172.21.15.53:6810/33158","heartbeat_front_addr":"172.21.15.53:6807/33158","state":["exists","up"]},{"osd":3,"uuid":"1a3a89c3-b449-46ba-bc65-3f53371d3af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":33159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":33159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6815","nonce":33159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":33159}]},"public_addr":"172.21.15.53:6812/33159","cluster_addr":"172.21.15.53:6813/33159","heartbeat_back_addr":"172.21.15.53:6815/33159","heartbeat_front_addr":"172.21.15.53:6814/33159","state":["exists","up"]},{"osd":4,"uuid":"d969fe78-c8c8-4c03-a534-2173406b0ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":33348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":33348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6815","nonce":33348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":33348}]},"public_addr":"172.21.15.104:6812/33348","cluster_addr":"172.21.15.104:6813/33348","heartbeat_back_addr":"172.21.15.104:6815/33348","heartbeat_front_addr":"172.21.15.104:6814/33348","state":["exists","up"]},{"osd":5,"uuid":"51e17a2f-42d3-49a6-b71e-dc7e13877c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":33346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":33346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":33346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":33346}]},"public_addr":"172.21.15.104:6800/33346","cluster_addr":"172.21.15.104:6801/33346","heartbeat_back_addr":"172.21.15.104:6803/33346","heartbeat_front_addr":"172.21.15.104:6802/33346","state":["exists","up"]},{"osd":6,"uuid":"c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":33357}]},"public_addr":"172.21.15.104:6808/33357","cluster_addr":"172.21.15.104:6809/33357","heartbeat_back_addr":"172.21.15.104:6811/33357","heartbeat_front_addr":"172.21.15.104:6810/33357","state":["exists","up"]},{"osd":7,"uuid":"7029aab4-31cc-4219-b7a4-a185564953b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":33347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":33347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":33347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":33347}]},"public_addr":"172.21.15.104:6804/33347","cluster_addr":"172.21.15.104:6805/33347","heartbeat_back_addr":"172.21.15.104:6807/33347","heartbeat_front_addr":"172.21.15.104:6806/33347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.615102+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.701424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.692098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.724924+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.688475+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.598581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.722216+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.661189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 10 librados: watch_flush enter 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 10 librados: watch_flush exit 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 20 client.4245.objecter shutdown clearing up homeless session... 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 10 client.4245.objecter successfully canceled tick 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 >> v2:172.21.15.104:6816/32804 conn(0x7fb12806e7f0 msgr2=0x7fb128070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 --2- 172.21.15.53:0/2899596499 >> v2:172.21.15.104:6816/32804 conn(0x7fb12806e7f0 0x7fb128070ca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb12c035040 tx=0x7fb12c032000 comp rx=0 tx=0).stop 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 10 monclient: shutdown 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:19.819 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb13c06f0d0 msgr2=0x7fb13c139ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb13c06f0d0 0x7fb13c139ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb13000cc20 tx=0x7fb130001a60 comp rx=0 tx=0).stop 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 shutdown_connections 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 --2- 172.21.15.53:0/2899596499 >> v2:172.21.15.104:6816/32804 conn(0x7fb12806e7f0 0x7fb128070ca0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13c06f4a0 0x7fb13c13a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb13c06f0d0 0x7fb13c139ee0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 --2- 172.21.15.53:0/2899596499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb13c067ce0 0x7fb13c13d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.817+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 >> 172.21.15.53:0/2899596499 conn(0x7fb13c0b1fe0 msgr2=0x7fb13c0aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.818+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 shutdown_connections 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.818+0000 7fb14375d640 1 -- 172.21.15.53:0/2899596499 wait complete. 2024-09-15T02:43:19.820 INFO:teuthology.misc.health.smithi053.stderr:2024-09-15T02:43:19.818+0000 7fb14375d640 1 librados: shutdown 2024-09-15T02:43:19.828 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-15T02:43:19.828 INFO:tasks.ceph:Creating RBD pool 2024-09-15T02:43:19.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T02:43:19.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.922+0000 7fbd6fd14640 10 monclient: get_monmap_and_config 2024-09-15T02:43:19.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.923+0000 7fbd6fd14640 10 monclient: build_initial_monmap 2024-09-15T02:43:19.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.923+0000 7fbd6fd14640 10 monclient: monmap: 2024-09-15T02:43:19.924 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:19.923798+0000 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:19.923798+0000 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:19.925 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:19.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.923+0000 7fbd6fd14640 1 Processor -- start 2024-09-15T02:43:19.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.923+0000 7fbd6fd14640 1 -- start start 2024-09-15T02:43:19.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.923+0000 7fbd6fd14640 10 monclient: init 2024-09-15T02:43:19.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd680b0fa0 0x7fbd6813a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient(hunting): picked mon.noname-b con 0x7fbd680b0fa0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd680a5920 0x7fbd6813aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient(hunting): picked mon.noname-a con 0x7fbd680a5920 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd6813dd30 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd680aed10 con 0x7fbd6813dd30 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd680a6b20 con 0x7fbd680a5920 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6813e8c0 con 0x7fbd680b0fa0 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:19.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6fd14640 10 monclient(hunting): authenticate will time out at 736.494477s 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd680a5920 0x7fbd6813aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd680b0fa0 0x7fbd6813a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd680b0fa0 0x7fbd6813a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:46934/0 (socket says 172.21.15.53:46934) 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd680a5920 0x7fbd6813aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51514/0 (socket says 172.21.15.53:51514) 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6e2b1640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbd6813dd30 0x7fbd6813e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40674/0 (socket says 172.21.15.53:40674) 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 1 -- 172.21.15.53:0/4276747022 learned_addr learned my addr 172.21.15.53:0/4276747022 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): get_auth_request con 0x7fbd680b0fa0 auth_method 0 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 10 monclient(hunting): get_auth_request con 0x7fbd680a5920 auth_method 0 2024-09-15T02:43:19.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6d2af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 1 -- 172.21.15.53:0/4276747022 >> v2:172.21.15.53:3300/0 conn(0x7fbd680a5920 msgr2=0x7fbd6813aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 1 --2- 172.21.15.53:0/4276747022 >> v2:172.21.15.53:3300/0 conn(0x7fbd680a5920 0x7fbd6813aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 1 -- 172.21.15.53:0/4276747022 >> v1:172.21.15.53:6789/0 conn(0x7fbd6813dd30 legacy=0x7fbd6813e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 10 monclient: _finish_hunting 0 2024-09-15T02:43:19.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6dab0640 1 monclient: found mon.noname-b 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6dab0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6dab0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6dab0640 1 -- 172.21.15.53:0/4276747022 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd680b1370 con 0x7fbd680b0fa0 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.924+0000 7fbd6caae640 1 -- 172.21.15.53:0/4276747022 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614614335 0 0) 0x7fbd680aed10 con 0x7fbd6813dd30 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6d2af640 1 --2- 172.21.15.53:0/4276747022 >> v2:172.21.15.53:3300/0 conn(0x7fbd680a5920 0x7fbd6813aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6dab0640 1 --2- 172.21.15.53:0/4276747022 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd680b0fa0 0x7fbd6813a9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009910 tx=0x7fbd600091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=898ebe57ceac246f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 1 -- 172.21.15.53:0/4276747022 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002add0 con 0x7fbd680b0fa0 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:19.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: dump: 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: _finish_auth 0 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:19.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 1 -- 172.21.15.53:0/4276747022 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd60032440 con 0x7fbd680b0fa0 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6fd14640 5 monclient: authenticate success, global_id 4223 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6fd14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6fd14640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6fd14640 4 set_mon_vals no callback set 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 1 -- 172.21.15.53:0/4276747022 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd600327e0 con 0x7fbd680b0fa0 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.925+0000 7fbd6caae640 10 monclient: dump: 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.938 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 10 monclient: shutdown 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 -- 172.21.15.53:0/4276747022 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd680b0fa0 msgr2=0x7fbd6813a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/4276747022 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd680b0fa0 0x7fbd6813a9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009910 tx=0x7fbd600091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 -- 172.21.15.53:0/4276747022 shutdown_connections 2024-09-15T02:43:19.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/4276747022 >> v2:172.21.15.53:3300/0 conn(0x7fbd680a5920 0x7fbd6813aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/4276747022 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd680b0fa0 0x7fbd6813a9b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 -- 172.21.15.53:0/4276747022 >> 172.21.15.53:0/4276747022 conn(0x7fbd68062d90 msgr2=0x7fbd68063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 -- 172.21.15.53:0/4276747022 shutdown_connections 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.926+0000 7fbd6fd14640 1 -- 172.21.15.53:0/4276747022 wait complete. 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 10 monclient: build_initial_monmap 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 10 monclient: monmap: 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:19.940 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 librados: starting msgr at 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 librados: starting objecter 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 Processor -- start 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 -- start start 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 librados: setting wanted keys 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 librados: calling monclient init 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 10 monclient: init 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:43:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd680a5920 0x7fbd680a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 10 monclient(hunting): picked mon.noname-b con 0x7fbd680a5920 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.927+0000 7fbd6fd14640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd680b0fa0 0x7fbd681416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient(hunting): picked mon.noname-a con 0x7fbd680b0fa0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd6813dd30 0x7fbd68141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient(hunting): picked mon.noname-c con 0x7fbd6813dd30 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68141230 con 0x7fbd680b0fa0 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68140f00 con 0x7fbd680a5920 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd680a8dc0 con 0x7fbd6813dd30 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient(hunting): authenticate will time out at 736.498040s 2024-09-15T02:43:19.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd680b0fa0 0x7fbd681416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd680b0fa0 0x7fbd681416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51518/0 (socket says 172.21.15.53:51518) 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 -- 172.21.15.53:0/2562592912 learned_addr learned my addr 172.21.15.53:0/2562592912 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd6813dd30 0x7fbd68141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): get_auth_request con 0x7fbd680b0fa0 auth_method 0 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 10 monclient(hunting): get_auth_request con 0x7fbd6813dd30 auth_method 0 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6e2b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6dab0640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd680a5920 0x7fbd680a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6dab0640 10 monclient(hunting): get_auth_request con 0x7fbd680a5920 auth_method 0 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6dab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6dab0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6dab0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 -- 172.21.15.53:0/2562592912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd6813dd30 msgr2=0x7fbd68141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd6813dd30 0x7fbd68141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 -- 172.21.15.53:0/2562592912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd680a5920 msgr2=0x7fbd680a7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd680a5920 0x7fbd680a7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient: _finish_hunting 0 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 monclient: found mon.noname-a 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 -- 172.21.15.53:0/2562592912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009390 con 0x7fbd680b0fa0 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6d2af640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd680b0fa0 0x7fbd681416c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00ec50 tx=0x7fbd5c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 client.?.objecter ms_handle_connect 0x7fbd680b0fa0 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 1 -- 172.21.15.53:0/2562592912 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c00ee90 con 0x7fbd680b0fa0 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:19.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: dump: 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 5 monclient: authenticate success, global_id 4254 2024-09-15T02:43:19.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 1 -- 172.21.15.53:0/2562592912 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c00cbc0 con 0x7fbd680b0fa0 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient: _renew_subs 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd60009450 con 0x7fbd680b0fa0 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 1 -- 172.21.15.53:0/2562592912 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c013470 con 0x7fbd680b0fa0 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd65ffb640 10 monclient: dump: 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:19.947 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.928+0000 7fbd67fff640 4 set_mon_vals no callback set 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.929+0000 7fbd6fd14640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.929+0000 7fbd6fd14640 10 monclient: _renew_subs 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.929+0000 7fbd6fd14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.929+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd681422e0 con 0x7fbd680b0fa0 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.929+0000 7fbd6fd14640 1 librados: init done 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.930+0000 7fbd6fd14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.930+0000 7fbd6fd14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.930+0000 7fbd6fd14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:19.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.930+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd681413a0 con 0x7fbd680b0fa0 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.930+0000 7fbd65ffb640 1 -- 172.21.15.53:0/2562592912 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 89555+0+0 (secure 0 0 0) 0x7fbd5c0137e0 con 0x7fbd680b0fa0 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd65ffb640 1 --2- 172.21.15.53:0/2562592912 >> v2:172.21.15.104:6816/32804 conn(0x7fbd5406e7f0 0x7fbd54070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd65ffb640 1 -- 172.21.15.53:0/2562592912 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3722+0+0 (secure 0 0 0) 0x7fbd5c08b790 con 0x7fbd680b0fa0 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd65ffb640 10 client.4254.objecter ms_dispatch 0x7fbd68000b90 osd_map(14..14 src has 1..14) v4 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd65ffb640 3 client.4254.objecter handle_osd_map got epochs [14,14] > 0 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd65ffb640 3 client.4254.objecter handle_osd_map decoding full epoch 14 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd65ffb640 20 client.4254.objecter dump_active .. 0 homeless 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd6dab0640 1 --2- 172.21.15.53:0/2562592912 >> v2:172.21.15.104:6816/32804 conn(0x7fbd5406e7f0 0x7fbd54070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd6dab0640 10 monclient: get_auth_request con 0x7fbd5406e7f0 auth_method 0 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd6dab0640 1 --2- 172.21.15.53:0/2562592912 >> v2:172.21.15.104:6816/32804 conn(0x7fbd5406e7f0 0x7fbd54070ca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbd60034000 tx=0x7fbd60032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:19.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.931+0000 7fbd65ffb640 10 client.4254.objecter ms_handle_connect 0x7fbd5406e7f0 2024-09-15T02:43:19.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.934+0000 7fbd65ffb640 1 -- 172.21.15.53:0/2562592912 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbd5c056050 con 0x7fbd680b0fa0 2024-09-15T02:43:19.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.934+0000 7fbd65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:19.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:19.934+0000 7fbd65ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.095+0000 7fbd6fd14640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T02:43:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.095+0000 7fbd6fd14640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T02:43:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.095+0000 7fbd6fd14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.095+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fbd680646e0 con 0x7fbd680b0fa0 2024-09-15T02:43:20.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.383+0000 7fbd65ffb640 1 -- 172.21.15.53:0/2562592912 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7fbd5c059860 con 0x7fbd680b0fa0 2024-09-15T02:43:20.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.383+0000 7fbd65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T02:43:20.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.383+0000 7fbd65ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-15T02:43:20.385 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'rbd' created 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 10 librados: watch_flush enter 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 10 librados: watch_flush exit 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 20 client.4254.objecter shutdown clearing up homeless session... 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 10 client.4254.objecter successfully canceled tick 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 >> v2:172.21.15.104:6816/32804 conn(0x7fbd5406e7f0 msgr2=0x7fbd54070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/2562592912 >> v2:172.21.15.104:6816/32804 conn(0x7fbd5406e7f0 0x7fbd54070ca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbd60034000 tx=0x7fbd60032000 comp rx=0 tx=0).stop 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 10 monclient: shutdown 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd680b0fa0 msgr2=0x7fbd681416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd680b0fa0 0x7fbd681416c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00ec50 tx=0x7fbd5c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 shutdown_connections 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd6813dd30 0x7fbd68141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd680b0fa0 0x7fbd681416c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/2562592912 >> v2:172.21.15.104:6816/32804 conn(0x7fbd5406e7f0 0x7fbd54070ca0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 --2- 172.21.15.53:0/2562592912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd680a5920 0x7fbd680a7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 >> 172.21.15.53:0/2562592912 conn(0x7fbd68062d90 msgr2=0x7fbd680af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 shutdown_connections 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 -- 172.21.15.53:0/2562592912 wait complete. 2024-09-15T02:43:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.386+0000 7fbd6fd14640 1 librados: shutdown 2024-09-15T02:43:20.399 DEBUG:teuthology.orchestra.run.smithi053:> rbd --cluster ceph pool init rbd 2024-09-15T02:43:20.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.443+0000 7f9c5ac29b80 10 monclient: get_monmap_and_config 2024-09-15T02:43:20.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.443+0000 7f9c5ac29b80 10 monclient: build_initial_monmap 2024-09-15T02:43:20.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.443+0000 7f9c5ac29b80 10 monclient: monmap: 2024-09-15T02:43:20.444 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:20.444 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:20.444 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:20.443888+0000 2024-09-15T02:43:20.444 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:20.443888+0000 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:20.445 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:20.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.443+0000 7f9c5ac29b80 1 Processor -- start 2024-09-15T02:43:20.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.443+0000 7f9c5ac29b80 1 -- start start 2024-09-15T02:43:20.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.443+0000 7f9c5ac29b80 10 monclient: init 2024-09-15T02:43:20.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient: _reopen_session rank -1 2024-09-15T02:43:20.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T02:43:20.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 1 --2- >> v2:172.21.15.53:3300/0 conn(0x55ab09d1cdc0 0x55ab09d1d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:20.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient(hunting): picked mon.noname-a con 0x55ab09d1cdc0 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient(hunting): picked mon.noname-a-legacy con 0x55ab09db5d00 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient(hunting): picked mon.noname-c-legacy con 0x55ab09db6740 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ab09c390e0 con 0x55ab09db5d00 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ab09c12270 con 0x55ab09db6740 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x55ab09c72260 con 0x55ab09d1cdc0 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient(hunting): _renew_subs 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c5ac29b80 10 monclient(hunting): authenticate will time out at 737.014576s 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x55ab09d1cdc0 0x55ab09d1d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c537fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x55ab09db5d00 0x55ab09d272a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40686/0 (socket says 172.21.15.53:40686) 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c537fe640 1 -- 172.21.15.53:0/2827767134 learned_addr learned my addr 172.21.15.53:0/2827767134 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 10 monclient(hunting): get_auth_request con 0x55ab09d1cdc0 auth_method 0 2024-09-15T02:43:20.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.444+0000 7f9c53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 1 -- 172.21.15.53:0/2827767134 >> v1:172.21.15.104:6789/0 conn(0x55ab09db6740 legacy=0x55ab09d27cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 1 -- 172.21.15.53:0/2827767134 >> v1:172.21.15.53:6789/0 conn(0x55ab09db5d00 legacy=0x55ab09d272a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 1 monclient: found mon.noname-a 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:20.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.445+0000 7f9c53fff640 1 -- 172.21.15.53:0/2827767134 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab09c1f0e0 con 0x55ab09d1cdc0 2024-09-15T02:43:20.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.446+0000 7f9c53fff640 1 --2- 172.21.15.53:0/2827767134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d1cdc0 0x55ab09d1d190 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400a5d0 tx=0x7f9c44045e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d656fee87d61a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:20.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 1 -- 172.21.15.53:0/2827767134 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4404d610 con 0x55ab09d1cdc0 2024-09-15T02:43:20.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:20.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: dump: 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: _finish_auth 0 2024-09-15T02:43:20.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 1 -- 172.21.15.53:0/2827767134 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4404d770 con 0x55ab09d1cdc0 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 1 -- 172.21.15.53:0/2827767134 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4404d900 con 0x55ab09d1cdc0 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 5 monclient: authenticate success, global_id 4263 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 10 monclient: get_monmap_and_config success 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 4 set_mon_vals no callback set 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c52ffd640 10 monclient: dump: 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 10 monclient: shutdown 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/2827767134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d1cdc0 msgr2=0x55ab09d1d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 1 --2- 172.21.15.53:0/2827767134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d1cdc0 0x55ab09d1d190 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400a5d0 tx=0x7f9c44045e00 comp rx=0 tx=0).stop 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/2827767134 shutdown_connections 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 1 --2- 172.21.15.53:0/2827767134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d1cdc0 0x55ab09d1d190 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/2827767134 >> 172.21.15.53:0/2827767134 conn(0x55ab09cfcbe0 msgr2=0x55ab09d2c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/2827767134 shutdown_connections 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.447+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/2827767134 wait complete. 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 10 monclient: build_initial_monmap 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 10 monclient: monmap: 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:20.459 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 1 librados: starting msgr at 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 1 librados: starting objecter 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 1 Processor -- start 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 1 -- start start 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 1 librados: setting wanted keys 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 1 librados: calling monclient init 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.448+0000 7f9c5ac29b80 10 monclient: init 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient: _reopen_session rank -1 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ab09db5d00 0x55ab09dbcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): picked mon.noname-b con 0x55ab09db5d00 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:20.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ab09db6740 0x55ab09dc8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): picked mon.noname-c con 0x55ab09db6740 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d21890 0x55ab09d21cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): picked mon.noname-a con 0x55ab09d21890 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab09c6df40 con 0x55ab09d21890 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab09c30ba0 con 0x55ab09db5d00 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ab09c30840 con 0x55ab09db6740 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): _renew_subs 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c5ac29b80 10 monclient(hunting): authenticate will time out at 737.019150s 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ab09db6740 0x55ab09dc8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d21890 0x55ab09d21cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ab09db6740 0x55ab09dc8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:46940/0 (socket says 172.21.15.53:46940) 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 -- 172.21.15.53:0/764878729 learned_addr learned my addr 172.21.15.53:0/764878729 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): get_auth_request con 0x55ab09db6740 auth_method 0 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 10 monclient(hunting): get_auth_request con 0x55ab09d21890 auth_method 0 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 1 --2- 172.21.15.53:0/764878729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ab09db5d00 0x55ab09dbcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 10 monclient(hunting): get_auth_request con 0x55ab09db5d00 auth_method 0 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c58f4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 -- 172.21.15.53:0/764878729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ab09db5d00 msgr2=0x55ab09dbcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 --2- 172.21.15.53:0/764878729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ab09db5d00 0x55ab09dbcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 -- 172.21.15.53:0/764878729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d21890 msgr2=0x55ab09d21cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 --2- 172.21.15.53:0/764878729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ab09d21890 0x55ab09d21cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient: _finish_hunting 0 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 monclient: found mon.noname-c 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4400a0e0 con 0x55ab09db6740 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c537fe640 1 --2- 172.21.15.53:0/764878729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ab09db6740 0x55ab09dc8060 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800bde0 tx=0x7f9c480017e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c43fff640 10 client.?.objecter ms_handle_connect 0x55ab09db6740 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c43fff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c48009e90 con 0x55ab09db6740 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.449+0000 7f9c43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 10 monclient: dump: 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 10 monclient: _finish_auth 0 2024-09-15T02:43:20.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4800e470 con 0x55ab09db6740 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 5 monclient: authenticate success, global_id 4235 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 10 monclient: _renew_subs 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ab09c12600 con 0x55ab09db6740 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4800e850 con 0x55ab09db6740 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c43fff640 10 monclient: dump: 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.466 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c527fc640 4 set_mon_vals no callback set 2024-09-15T02:43:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 10 monclient: _renew_subs 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab09c12060 con 0x55ab09db6740 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 1 librados: init done 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.450+0000 7f9c5ac29b80 10 librados: wait_for_osdmap waiting 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.451+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 4) v1 ==== 89555+0+0 (secure 0 0 0) 0x7f9c4800ec80 con 0x55ab09db6740 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.451+0000 7f9c43fff640 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6816/32804 conn(0x7f9c3806c910 0x7f9c3806edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.451+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4075+0+0 (secure 0 0 0) 0x7f9c4808a5a0 con 0x55ab09db6740 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.451+0000 7f9c43fff640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_map(15..15 src has 1..15) v4 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.451+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map got epochs [15,15] > 0 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.451+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map decoding full epoch 15 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c53fff640 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6816/32804 conn(0x7f9c3806c910 0x7f9c3806edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c53fff640 10 monclient: get_auth_request con 0x7f9c3806c910 auth_method 0 2024-09-15T02:43:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c43fff640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-15T02:43:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c5ac29b80 10 librados: wait_for_osdmap done waiting 2024-09-15T02:43:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c5ac29b80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T02:43:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c5ac29b80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T02:43:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c5ac29b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x55ab09d22410 con 0x55ab09db6740 2024-09-15T02:43:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c43fff640 10 client.4235.objecter ms_handle_connect 0x7f9c3806c910 2024-09-15T02:43:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:20.452+0000 7f9c53fff640 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6816/32804 conn(0x7f9c3806c910 0x7f9c3806edc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c44057000 tx=0x7f9c44055000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:21.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.405+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9c4801c0a0 con 0x55ab09db6740 2024-09-15T02:43:21.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.405+0000 7f9c43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T02:43:21.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.405+0000 7f9c43fff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-15T02:43:21.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.405+0000 7f9c5ac29b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:21.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.405+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55ab09c30630 con 0x55ab09db6740 2024-09-15T02:43:21.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.409+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c480892c0 con 0x55ab09db6740 2024-09-15T02:43:21.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.409+0000 7f9c43fff640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T02:43:21.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.409+0000 7f9c527fc640 10 client.4235.objecter _get_latest_version latest 16, waiting 2024-09-15T02:43:21.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.409+0000 7f9c527fc640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:21.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.409+0000 7f9c527fc640 10 monclient: _renew_subs 2024-09-15T02:43:21.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.409+0000 7f9c527fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:21.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.409+0000 7f9c527fc640 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=16}) v3 -- 0x7f9c30003b40 con 0x55ab09db6740 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 1028+0+0 (secure 0 0 0) 0x7f9c4805e0d0 con 0x55ab09db6740 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c43fff640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_map(16..16 src has 1..16) v4 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map got epochs [16,16] > 15 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map decoding incremental epoch 16 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c43fff640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c5ac29b80 10 librados: create oid=rbd_trash nspace= 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c5ac29b80 10 client.4235.objecter _op_submit op 0x55ab09d1f840 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c5ac29b80 20 client.4235.objecter _calc_target epoch 16 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c5ac29b80 20 client.4235.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c5ac29b80 10 client.4235.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-15T02:43:21.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c5ac29b80 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6800/33346 conn(0x55ab09d20240 0x55ab09db2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.501+0000 7f9c5ac29b80 20 client.4235.objecter _get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c5ac29b80 10 client.4235.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c5ac29b80 20 client.4235.objecter get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c5ac29b80 15 client.4235.objecter _session_op_assign 5 1 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c5ac29b80 15 client.4235.objecter _send_op 1 to 2.3 on osd.5 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 --> v2:172.21.15.104:6800/33346 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x55ab09db2bc0 con 0x55ab09d20240 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c5ac29b80 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c5ac29b80 5 client.4235.objecter 1 in flight 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c58f4e640 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6800/33346 conn(0x55ab09d20240 0x55ab09db2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c58f4e640 10 monclient: get_auth_request con 0x55ab09d20240 auth_method 0 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c43fff640 10 client.4235.objecter ms_handle_connect 0x55ab09d20240 2024-09-15T02:43:21.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.502+0000 7f9c58f4e640 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6800/33346 conn(0x55ab09d20240 0x55ab09db2680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 1 -- 172.21.15.53:0/764878729 <== osd.5 v2:172.21.15.104:6800/33346 1 ==== osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f9c4c00e6a0 con 0x55ab09d20240 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) v8 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 10 client.4235.objecter in handle_osd_op_reply 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 7 client.4235.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 15 client.4235.objecter handle_osd_op_reply completed tid 1 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 15 client.4235.objecter _finish_op 1 2024-09-15T02:43:21.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:21.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 15 client.4235.objecter _session_op_remove 5 1 2024-09-15T02:43:21.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c58f4e640 5 client.4235.objecter 0 in flight 2024-09-15T02:43:21.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.504+0000 7f9c5ac29b80 10 librados: Objecter returned from create r=0 2024-09-15T02:43:21.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 10 client.4235.objecter _op_submit op 0x55ab09e1b6b0 2024-09-15T02:43:21.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 20 client.4235.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:43:21.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 20 client.4235.objecter _get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 20 client.4235.objecter get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 15 client.4235.objecter _session_op_assign 5 2 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 15 client.4235.objecter _send_op 2 to 2.2 on osd.5 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 --> v2:172.21.15.104:6800/33346 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) v8 -- 0x55ab09e1bc40 con 0x55ab09d20240 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 5 client.4235.objecter 1 in flight 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 1 -- 172.21.15.53:0/764878729 <== osd.5 v2:172.21.15.104:6800/33346 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9c4c00e6a0 con 0x55ab09d20240 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 10 client.4235.objecter in handle_osd_op_reply 2024-09-15T02:43:21.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 7 client.4235.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 10 client.4235.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 15 client.4235.objecter handle_osd_op_reply completed tid 2 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 15 client.4235.objecter _finish_op 2 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 15 client.4235.objecter _session_op_remove 5 2 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c58f4e640 5 client.4235.objecter 0 in flight 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.506+0000 7f9c5ac29b80 10 client.4235.objecter _op_submit op 0x55ab09e1a840 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 20 client.4235.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 20 client.4235.objecter _get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 20 client.4235.objecter get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 15 client.4235.objecter _session_op_assign 5 3 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 15 client.4235.objecter _send_op 3 to 2.2 on osd.5 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 --> v2:172.21.15.104:6800/33346 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) v8 -- 0x55ab09e1ad30 con 0x55ab09d20240 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c5ac29b80 5 client.4235.objecter 1 in flight 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 1 -- 172.21.15.53:0/764878729 <== osd.5 v2:172.21.15.104:6800/33346 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9c4c00e6a0 con 0x55ab09d20240 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 10 client.4235.objecter in handle_osd_op_reply 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 7 client.4235.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 10 client.4235.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 15 client.4235.objecter handle_osd_op_reply completed tid 3 2024-09-15T02:43:21.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 15 client.4235.objecter _finish_op 3 2024-09-15T02:43:21.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:21.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 15 client.4235.objecter _session_op_remove 5 3 2024-09-15T02:43:21.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c58f4e640 5 client.4235.objecter 0 in flight 2024-09-15T02:43:21.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c51ffb640 10 client.4235.objecter allocate_selfmanaged_snap; pool: 2 2024-09-15T02:43:21.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c51ffb640 10 client.4235.objecter pool_op_submit 4 2024-09-15T02:43:21.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c51ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:21.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:21.507+0000 7f9c51ffb640 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f9c340030b0 con 0x55ab09db6740 2024-09-15T02:43:22.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.025+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 9 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f9c48054850 con 0x55ab09db6740 2024-09-15T02:43:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 10 ==== pool_op_reply(tid 4 (0) Success v17) v1 ==== 55+0+0 (secure 0 0 0) 0x7f9c48054da0 con 0x55ab09db6740 2024-09-15T02:43:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 pool_op_reply(tid 4 (0) Success v17) v1 2024-09-15T02:43:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 client.4235.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v17) v1 2024-09-15T02:43:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 client.4235.objecter have request 4 at 0x55ab09d239d0 Op: create unmanaged snap 2024-09-15T02:43:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 20 client.4235.objecter waiting for client to reach epoch 17 before calling back 2024-09-15T02:43:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 monclient: _renew_subs 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=17}) v3 -- 0x7f9c380292c0 con 0x55ab09db6740 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 client.4235.objecter done 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 11 ==== osd_map(17..17 src has 1..17) v4 ==== 618+0+0 (secure 0 0 0) 0x7f9c48058950 con 0x55ab09db6740 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_map(17..17 src has 1..17) v4 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map got epochs [17,17] > 16 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map decoding incremental epoch 17 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c43fff640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 20 librados: queue_aio_write 0x55ab09e169f0 completion 0x55ab09c66a70 write_seq 1 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 10 client.4235.objecter _op_submit op 0x7f9c30006110 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 20 client.4235.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T02:43:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 20 client.4235.objecter _get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:22.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-15T02:43:22.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 20 client.4235.objecter get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:22.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 15 client.4235.objecter _session_op_assign 5 5 2024-09-15T02:43:22.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 15 client.4235.objecter _send_op 5 to 2.2 on osd.5 2024-09-15T02:43:22.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 1 -- 172.21.15.53:0/764878729 --> v2:172.21.15.104:6800/33346 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) v8 -- 0x7f9c30006600 con 0x55ab09d20240 2024-09-15T02:43:22.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:22.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.411+0000 7f9c527fc640 5 client.4235.objecter 1 in flight 2024-09-15T02:43:22.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 1 -- 172.21.15.53:0/764878729 <== osd.5 v2:172.21.15.104:6800/33346 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f9c4c00e6a0 con 0x55ab09d20240 2024-09-15T02:43:22.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) v8 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 10 client.4235.objecter in handle_osd_op_reply 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 7 client.4235.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 10 client.4235.objecter op 1 rval 0 len 0 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 15 client.4235.objecter handle_osd_op_reply completed tid 5 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 15 client.4235.objecter _finish_op 5 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 15 client.4235.objecter _session_op_remove 5 5 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 5 client.4235.objecter 0 in flight 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c58f4e640 20 librados: complete_aio_write 0x55ab09c66a70 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c51ffb640 10 client.4235.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c51ffb640 10 client.4235.objecter pool_op_submit 6 2024-09-15T02:43:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c51ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:22.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:22.416+0000 7f9c51ffb640 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v17) v4 -- 0x7f9c34004530 con 0x55ab09db6740 2024-09-15T02:43:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 12 ==== pool_op_reply(tid 6 (0) Success v18) v1 ==== 43+0+0 (secure 0 0 0) 0x7f9c48054bd0 con 0x55ab09db6740 2024-09-15T02:43:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 pool_op_reply(tid 6 (0) Success v18) v1 2024-09-15T02:43:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 client.4235.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v18) v1 2024-09-15T02:43:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 client.4235.objecter have request 6 at 0x7f9c340044d0 Op: delete unmanaged snap 2024-09-15T02:43:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 20 client.4235.objecter waiting for client to reach epoch 18 before calling back 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 monclient: _renew_subs 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=18}) v3 -- 0x7f9c38029450 con 0x55ab09db6740 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 client.4235.objecter done 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 1 -- 172.21.15.53:0/764878729 <== mon.2 v2:172.21.15.53:3301/0 13 ==== osd_map(18..18 src has 1..18) v4 ==== 646+0+0 (secure 0 0 0) 0x7f9c4807f650 con 0x55ab09db6740 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map got epochs [18,18] > 17 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 3 client.4235.objecter handle_osd_map decoding incremental epoch 18 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.420+0000 7f9c43fff640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 20 librados: queue_aio_write 0x55ab09e169f0 completion 0x7f9c34001d10 write_seq 2 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 10 client.4235.objecter _op_submit op 0x7f9c30006110 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 20 client.4235.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:43:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 20 client.4235.objecter _get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 20 client.4235.objecter get_session s=0x55ab09c3ee60 osd=5 3 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 15 client.4235.objecter _session_op_assign 5 7 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 15 client.4235.objecter _send_op 7 to 2.2 on osd.5 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 1 -- 172.21.15.53:0/764878729 --> v2:172.21.15.104:6800/33346 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x7f9c30006a30 con 0x55ab09d20240 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.421+0000 7f9c527fc640 5 client.4235.objecter 1 in flight 2024-09-15T02:43:23.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 1 -- 172.21.15.53:0/764878729 <== osd.5 v2:172.21.15.104:6800/33346 5 ==== osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9c4c00e6a0 con 0x55ab09d20240 2024-09-15T02:43:23.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 10 client.4235.objecter ms_dispatch 0x55ab09c89020 osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) v8 2024-09-15T02:43:23.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 10 client.4235.objecter in handle_osd_op_reply 2024-09-15T02:43:23.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 7 client.4235.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-15T02:43:23.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-15T02:43:23.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 15 client.4235.objecter handle_osd_op_reply completed tid 7 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 15 client.4235.objecter _finish_op 7 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 4 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 15 client.4235.objecter _session_op_remove 5 7 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 5 client.4235.objecter 0 in flight 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 20 librados: complete_aio_write 0x7f9c34001d10 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 10 librados: watch_flush enter 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 10 librados: watch_flush exit 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 10 client.4235.objecter close_session for osd.5 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6800/33346 conn(0x55ab09d20240 msgr2=0x55ab09db2680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6800/33346 conn(0x55ab09d20240 0x55ab09db2680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 20 client.4235.objecter put_session s=0x55ab09c3ee60 osd=5 1 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 20 client.4235.objecter shutdown clearing up homeless session... 2024-09-15T02:43:23.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 10 client.4235.objecter successfully canceled tick 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6816/32804 conn(0x7f9c3806c910 msgr2=0x7f9c3806edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 --2- 172.21.15.53:0/764878729 >> v2:172.21.15.104:6816/32804 conn(0x7f9c3806c910 0x7f9c3806edc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c44057000 tx=0x7f9c44055000 comp rx=0 tx=0).stop 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 10 monclient: shutdown 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ab09db6740 msgr2=0x55ab09dc8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 --2- 172.21.15.53:0/764878729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ab09db6740 0x55ab09dc8060 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800bde0 tx=0x7f9c480017e0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c58f4e640 1 -- 172.21.15.53:0/764878729 reap_dead start 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 shutdown_connections 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.424+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 >> 172.21.15.53:0/764878729 conn(0x55ab09d1cdc0 msgr2=0x55ab09d271a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.425+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 shutdown_connections 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.425+0000 7f9c5ac29b80 1 -- 172.21.15.53:0/764878729 wait complete. 2024-09-15T02:43:23.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.425+0000 7f9c5ac29b80 1 librados: shutdown 2024-09-15T02:43:23.430 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T02:43:23.431 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T02:43:23.431 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T02:43:23.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T02:43:23.431 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T02:43:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.536+0000 7fe08c55d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.536+0000 7fe08c55d640 10 monclient: build_initial_monmap 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.536+0000 7fe08c55d640 10 monclient: monmap: 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:23.537619+0000 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:23.537619+0000 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:23.539 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.537+0000 7fe08c55d640 1 Processor -- start 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.537+0000 7fe08c55d640 1 -- start start 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.537+0000 7fe08c55d640 10 monclient: init 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0840b0df0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0840b11c0 0x7fe0840b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0840b11c0 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe0840a56d0 0x7fe0840aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0840a56d0 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08413e400 con 0x7fe0840b0df0 2024-09-15T02:43:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0840b16d0 con 0x7fe0840b11c0 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0840b1810 con 0x7fe0840a56d0 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08c55d640 10 monclient(hunting): authenticate will time out at 740.108428s 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0840b11c0 0x7fe0840b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0840b11c0 0x7fe0840b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51550/0 (socket says 172.21.15.53:51550) 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 -- 172.21.15.53:0/3674867364 learned_addr learned my addr 172.21.15.53:0/3674867364 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): get_auth_request con 0x7fe0840b11c0 auth_method 0 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08aafa640 1 --2- 172.21.15.53:0/3674867364 >> v2:172.21.15.104:3300/0 conn(0x7fe0840a56d0 0x7fe0840aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08aafa640 10 monclient(hunting): get_auth_request con 0x7fe0840a56d0 auth_method 0 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08aafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08aafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe08aafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 -- 172.21.15.53:0/3674867364 >> v2:172.21.15.104:3300/0 conn(0x7fe0840a56d0 msgr2=0x7fe0840aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 --2- 172.21.15.53:0/3674867364 >> v2:172.21.15.104:3300/0 conn(0x7fe0840a56d0 0x7fe0840aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 -- 172.21.15.53:0/3674867364 >> v1:172.21.15.104:6789/0 conn(0x7fe0840b0df0 legacy=0x7fe0840afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient: _finish_hunting 0 2024-09-15T02:43:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 monclient: found mon.noname-a 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.538+0000 7fe089af8640 1 -- 172.21.15.53:0/3674867364 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08413dea0 con 0x7fe0840b11c0 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe089af8640 1 --2- 172.21.15.53:0/3674867364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840b11c0 0x7fe0840b03a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe0840a5e90 tx=0x7fe0740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be728714283d59f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 1 -- 172.21.15.53:0/3674867364 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07402ad50 con 0x7fe0840b11c0 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: dump: 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.550 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: _finish_auth 0 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 1 -- 172.21.15.53:0/3674867364 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe074032440 con 0x7fe0840b11c0 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 1 -- 172.21.15.53:0/3674867364 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074032820 con 0x7fe0840b11c0 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe08c55d640 5 monclient: authenticate success, global_id 4275 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe08c55d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe08c55d640 4 set_mon_vals no callback set 2024-09-15T02:43:23.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.539+0000 7fe0892f7640 10 monclient: dump: 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 10 monclient: shutdown 2024-09-15T02:43:23.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 -- 172.21.15.53:0/3674867364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840b11c0 msgr2=0x7fe0840b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 --2- 172.21.15.53:0/3674867364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840b11c0 0x7fe0840b03a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe0840a5e90 tx=0x7fe0740091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 -- 172.21.15.53:0/3674867364 shutdown_connections 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 --2- 172.21.15.53:0/3674867364 >> v2:172.21.15.104:3300/0 conn(0x7fe0840a56d0 0x7fe0840aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 --2- 172.21.15.53:0/3674867364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840b11c0 0x7fe0840b03a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 -- 172.21.15.53:0/3674867364 >> 172.21.15.53:0/3674867364 conn(0x7fe084062d90 msgr2=0x7fe084063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 -- 172.21.15.53:0/3674867364 shutdown_connections 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.540+0000 7fe08c55d640 1 -- 172.21.15.53:0/3674867364 wait complete. 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 10 monclient: build_initial_monmap 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 10 monclient: monmap: 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:23.553 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 1 librados: starting msgr at 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 1 librados: starting objecter 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 1 Processor -- start 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 1 -- start start 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 1 librados: setting wanted keys 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 1 librados: calling monclient init 2024-09-15T02:43:23.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.541+0000 7fe08c55d640 10 monclient: init 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840a56d0 0x7fe0841414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0840a56d0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0840b0df0 0x7fe084150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0840b0df0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0840b11c0 0x7fe0840a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0840b11c0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840a99e0 con 0x7fe0840a56d0 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840aafc0 con 0x7fe0840b0df0 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840a9680 con 0x7fe0840b11c0 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08c55d640 10 monclient(hunting): authenticate will time out at 740.112264s 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840a56d0 0x7fe0841414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840a56d0 0x7fe0841414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51558/0 (socket says 172.21.15.53:51558) 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 -- 172.21.15.53:0/617560261 learned_addr learned my addr 172.21.15.53:0/617560261 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): get_auth_request con 0x7fe0840a56d0 auth_method 0 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0840b11c0 0x7fe0840a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 10 monclient(hunting): get_auth_request con 0x7fe0840b11c0 auth_method 0 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08aafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe089af8640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0840b0df0 0x7fe084150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe089af8640 10 monclient(hunting): get_auth_request con 0x7fe0840b0df0 auth_method 0 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe089af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe089af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe089af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 -- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0840b11c0 msgr2=0x7fe0840a6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0840b11c0 0x7fe0840a6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 -- 172.21.15.53:0/617560261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0840b0df0 msgr2=0x7fe084150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0840b0df0 0x7fe084150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient: _finish_hunting 0 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 monclient: found mon.noname-a 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 -- 172.21.15.53:0/617560261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe074009450 con 0x7fe0840a56d0 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe089af8640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0840b0df0 0x7fe084150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe08a2f9640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840a56d0 0x7fe0841414b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe08000c610 tx=0x7fe080007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe07a7fc640 10 client.?.objecter ms_handle_connect 0x7fe0840a56d0 2024-09-15T02:43:23.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.542+0000 7fe07a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 1 -- 172.21.15.53:0/617560261 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080012630 con 0x7fe0840a56d0 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: dump: 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.559 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 1 -- 172.21.15.53:0/617560261 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0800127b0 con 0x7fe0840a56d0 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 5 monclient: authenticate success, global_id 4278 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 10 monclient: _renew_subs 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0840ab290 con 0x7fe0840a56d0 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 1 -- 172.21.15.53:0/617560261 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080012b70 con 0x7fe0840a56d0 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe07a7fc640 10 monclient: dump: 2024-09-15T02:43:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe088af6640 4 set_mon_vals no callback set 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 10 monclient: _renew_subs 2024-09-15T02:43:23.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0840a92c0 con 0x7fe0840a56d0 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.543+0000 7fe08c55d640 1 librados: init done 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.544+0000 7fe08c55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.544+0000 7fe08c55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.544+0000 7fe08c55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.544+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0840ab100 con 0x7fe0840a56d0 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.544+0000 7fe07a7fc640 1 -- 172.21.15.53:0/617560261 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fe08001a3b0 con 0x7fe0840a56d0 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe07a7fc640 1 --2- 172.21.15.53:0/617560261 >> v2:172.21.15.104:6816/32804 conn(0x7fe06806e800 0x7fe068070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe07a7fc640 1 -- 172.21.15.53:0/617560261 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7fe08008b260 con 0x7fe0840a56d0 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe07a7fc640 10 client.4278.objecter ms_dispatch 0x7fe084000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe07a7fc640 3 client.4278.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:23.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe07a7fc640 3 client.4278.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe089af8640 1 --2- 172.21.15.53:0/617560261 >> v2:172.21.15.104:6816/32804 conn(0x7fe06806e800 0x7fe068070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe089af8640 10 monclient: get_auth_request con 0x7fe06806e800 auth_method 0 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.545+0000 7fe07a7fc640 20 client.4278.objecter dump_active .. 0 homeless 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.546+0000 7fe089af8640 1 --2- 172.21.15.53:0/617560261 >> v2:172.21.15.104:6816/32804 conn(0x7fe06806e800 0x7fe068070cb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe074034000 tx=0x7fe074032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.546+0000 7fe07a7fc640 10 client.4278.objecter ms_handle_connect 0x7fe06806e800 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.549+0000 7fe07a7fc640 1 -- 172.21.15.53:0/617560261 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe0800557f0 con 0x7fe0840a56d0 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.549+0000 7fe07a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:23.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.549+0000 7fe07a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:23.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.756+0000 7fe08c55d640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-15T02:43:23.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.756+0000 7fe08c55d640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T02:43:23.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.756+0000 7fe08c55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.756+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe0840646e0 con 0x7fe0840a56d0 2024-09-15T02:43:23.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.759+0000 7fe07a7fc640 1 -- 172.21.15.53:0/617560261 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+174082 (secure 0 0 0) 0x7fe080059000 con 0x7fe0840a56d0 2024-09-15T02:43:23.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.759+0000 7fe07a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T02:43:23.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.759+0000 7fe07a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:23.763 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:43:23.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 10 librados: watch_flush enter 2024-09-15T02:43:23.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 10 librados: watch_flush exit 2024-09-15T02:43:23.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 20 client.4278.objecter shutdown clearing up homeless session... 2024-09-15T02:43:23.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 10 client.4278.objecter successfully canceled tick 2024-09-15T02:43:23.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 >> v2:172.21.15.104:6816/32804 conn(0x7fe06806e800 msgr2=0x7fe068070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 --2- 172.21.15.53:0/617560261 >> v2:172.21.15.104:6816/32804 conn(0x7fe06806e800 0x7fe068070cb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe074034000 tx=0x7fe074032000 comp rx=0 tx=0).stop 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 10 monclient: shutdown 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840a56d0 msgr2=0x7fe0841414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840a56d0 0x7fe0841414b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe08000c610 tx=0x7fe080007ed0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 shutdown_connections 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0840b11c0 0x7fe0840a6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 --2- 172.21.15.53:0/617560261 >> v2:172.21.15.104:6816/32804 conn(0x7fe06806e800 0x7fe068070cb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0840b0df0 0x7fe084150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 --2- 172.21.15.53:0/617560261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0840a56d0 0x7fe0841414b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 >> 172.21.15.53:0/617560261 conn(0x7fe084062d90 msgr2=0x7fe0840b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.761+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 shutdown_connections 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.762+0000 7fe08c55d640 1 -- 172.21.15.53:0/617560261 wait complete. 2024-09-15T02:43:23.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.762+0000 7fe08c55d640 1 librados: shutdown 2024-09-15T02:43:23.773 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":32804}]},"active_addr":"172.21.15.104:6816/32804","active_change":"2024-09-15T02:43:13.324488+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4103,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":1625795713}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":1127078567}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":1440477178}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":3499264273}]}]} 2024-09-15T02:43:23.777 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T02:43:23.777 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T02:43:23.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:43:23.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.872+0000 7f4d33b00640 10 monclient: get_monmap_and_config 2024-09-15T02:43:23.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.872+0000 7f4d33b00640 10 monclient: build_initial_monmap 2024-09-15T02:43:23.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.872+0000 7f4d33b00640 10 monclient: monmap: 2024-09-15T02:43:23.873 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:23.873 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.873 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:23.873128+0000 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:23.873128+0000 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:23.874 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.872+0000 7f4d33b00640 1 Processor -- start 2024-09-15T02:43:23.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 1 -- start start 2024-09-15T02:43:23.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient: init 2024-09-15T02:43:23.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d2c06bae0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d2c06beb0 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c136f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient(hunting): picked mon.noname-b con 0x7f4d2c13a8e0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c139d40 con 0x7f4d2c06beb0 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c06cda0 con 0x7f4d2c06bae0 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c06d080 con 0x7f4d2c13a8e0 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d33b00640 10 monclient(hunting): authenticate will time out at 740.443812s 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d3209d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d3209d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:46950/0 (socket says 172.21.15.53:46950) 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d3209d640 1 -- 172.21.15.53:0/537384376 learned_addr learned my addr 172.21.15.53:0/537384376 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d3209d640 10 monclient(hunting): get_auth_request con 0x7f4d2c13a8e0 auth_method 0 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d3209d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d3209d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.873+0000 7f4d3209d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 1 -- 172.21.15.53:0/537384376 >> v1:172.21.15.104:6789/0 conn(0x7f4d2c06bae0 legacy=0x7f4d2c06b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 1 -- 172.21.15.53:0/537384376 >> v1:172.21.15.53:6789/0 conn(0x7f4d2c06beb0 legacy=0x7f4d2c13a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 1 monclient: found mon.noname-b 2024-09-15T02:43:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 1 -- 172.21.15.53:0/537384376 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c0aab50 con 0x7f4d2c13a8e0 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.874+0000 7f4d3209d640 1 --2- 172.21.15.53:0/537384376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c136f40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400d1d0 tx=0x7f4d2400ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d29839d0c59b189 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 1 -- 172.21.15.53:0/537384376 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2400b2e0 con 0x7f4d2c13a8e0 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: dump: 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.880 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: _finish_auth 0 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 1 -- 172.21.15.53:0/537384376 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2400b440 con 0x7f4d2c13a8e0 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 5 monclient: authenticate success, global_id 4244 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 1 -- 172.21.15.53:0/537384376 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24009840 con 0x7f4d2c13a8e0 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 4 set_mon_vals no callback set 2024-09-15T02:43:23.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d3089a640 10 monclient: dump: 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 10 monclient: shutdown 2024-09-15T02:43:23.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 1 -- 172.21.15.53:0/537384376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 msgr2=0x7f4d2c136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 1 --2- 172.21.15.53:0/537384376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c136f40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400d1d0 tx=0x7f4d2400ba50 comp rx=0 tx=0).stop 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 1 -- 172.21.15.53:0/537384376 shutdown_connections 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 1 --2- 172.21.15.53:0/537384376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c136f40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 1 -- 172.21.15.53:0/537384376 >> 172.21.15.53:0/537384376 conn(0x7f4d2c067b00 msgr2=0x7f4d2c0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 1 -- 172.21.15.53:0/537384376 shutdown_connections 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.875+0000 7f4d33b00640 1 -- 172.21.15.53:0/537384376 wait complete. 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 10 monclient: build_initial_monmap 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 10 monclient: monmap: 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 1 librados: starting msgr at 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 1 librados: starting objecter 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 1 Processor -- start 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 1 -- start start 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 1 librados: setting wanted keys 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 1 librados: calling monclient init 2024-09-15T02:43:23.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.876+0000 7f4d33b00640 10 monclient: init 2024-09-15T02:43:23.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:23.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:23.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d2c06bae0 0x7f4d2c13dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): picked mon.noname-a con 0x7f4d2c06bae0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d2c06beb0 0x7f4d2c0a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): picked mon.noname-b con 0x7f4d2c06beb0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): picked mon.noname-c con 0x7f4d2c13a8e0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c117e80 con 0x7f4d2c06bae0 2024-09-15T02:43:23.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c13c570 con 0x7f4d2c06beb0 2024-09-15T02:43:23.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:23.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c13d0b0 con 0x7f4d2c13a8e0 2024-09-15T02:43:23.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:23.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d33b00640 10 monclient(hunting): authenticate will time out at 740.447440s 2024-09-15T02:43:23.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d2c06bae0 0x7f4d2c13dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d2c06bae0 0x7f4d2c13dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51572/0 (socket says 172.21.15.53:51572) 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 -- 172.21.15.53:0/1082349647 learned_addr learned my addr 172.21.15.53:0/1082349647 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): get_auth_request con 0x7f4d2c06bae0 auth_method 0 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 10 monclient(hunting): get_auth_request con 0x7f4d2c13a8e0 auth_method 0 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:23.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:23.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:23.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:23.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d2c06beb0 0x7f4d2c0a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 10 monclient(hunting): get_auth_request con 0x7f4d2c06beb0 auth_method 0 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3109b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-15T02:43:23.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 -- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 msgr2=0x7f4d2c150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 -- 172.21.15.53:0/1082349647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d2c06beb0 msgr2=0x7f4d2c0a9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d2c06beb0 0x7f4d2c0a9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient: _finish_hunting 0 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 monclient: found mon.noname-a 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3189c640 1 -- 172.21.15.53:0/1082349647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2400bcf0 con 0x7f4d2c06bae0 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.877+0000 7f4d3209d640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d3189c640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d2c06bae0 0x7f4d2c13dc80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800b520 tx=0x7f4d280090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 client.?.objecter ms_handle_connect 0x7f4d2c06bae0 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 1 -- 172.21.15.53:0/1082349647 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800f510 con 0x7f4d2c06bae0 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: dump: 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.897 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 1 -- 172.21.15.53:0/1082349647 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2800f670 con 0x7f4d2c06bae0 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 5 monclient: authenticate success, global_id 4284 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 10 monclient: _renew_subs 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c13d330 con 0x7f4d2c06bae0 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 1 -- 172.21.15.53:0/1082349647 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800fa30 con 0x7f4d2c06bae0 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d21ffb640 10 monclient: dump: 2024-09-15T02:43:23.898 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d23fff640 4 set_mon_vals no callback set 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 10 monclient: _renew_subs 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c1513a0 con 0x7f4d2c06bae0 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.878+0000 7f4d33b00640 1 librados: init done 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.879+0000 7f4d33b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.879+0000 7f4d33b00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.879+0000 7f4d33b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.879+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c0646e0 con 0x7f4d2c06bae0 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.879+0000 7f4d21ffb640 1 -- 172.21.15.53:0/1082349647 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f4d2801a400 con 0x7f4d2c06bae0 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d21ffb640 1 --2- 172.21.15.53:0/1082349647 >> v2:172.21.15.104:6816/32804 conn(0x7f4d1006e880 0x7f4d10070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d21ffb640 1 -- 172.21.15.53:0/1082349647 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f4d2808b0f0 con 0x7f4d2c06bae0 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d21ffb640 10 client.4284.objecter ms_dispatch 0x7f4d2c000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d21ffb640 3 client.4284.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d21ffb640 3 client.4284.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d3109b640 1 --2- 172.21.15.53:0/1082349647 >> v2:172.21.15.104:6816/32804 conn(0x7f4d1006e880 0x7f4d10070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:23.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d21ffb640 20 client.4284.objecter dump_active .. 0 homeless 2024-09-15T02:43:23.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d3109b640 10 monclient: get_auth_request con 0x7f4d1006e880 auth_method 0 2024-09-15T02:43:23.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d3109b640 1 --2- 172.21.15.53:0/1082349647 >> v2:172.21.15.104:6816/32804 conn(0x7f4d1006e880 0x7f4d10070d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c118ec0 tx=0x7f4d1c007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:23.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.880+0000 7f4d21ffb640 10 client.4284.objecter ms_handle_connect 0x7f4d1006e880 2024-09-15T02:43:23.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.884+0000 7f4d21ffb640 1 -- 172.21.15.53:0/1082349647 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4d280557b0 con 0x7f4d2c06bae0 2024-09-15T02:43:23.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.884+0000 7f4d21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:23.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:23.884+0000 7f4d21ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:24.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.048+0000 7f4d33b00640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:24.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.048+0000 7f4d33b00640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:24.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.048+0000 7f4d33b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.048+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d2c136d20 con 0x7f4d2c06bae0 2024-09-15T02:43:24.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.049+0000 7f4d21ffb640 1 -- 172.21.15.53:0/1082349647 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+11979 (secure 0 0 0) 0x7f4d28058fc0 con 0x7f4d2c06bae0 2024-09-15T02:43:24.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.049+0000 7f4d21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:24.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.049+0000 7f4d21ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:24.051 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:43:24.051 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":18,"fsid":"7e7cb607-2b58-4a36-a082-99425792b70f","created":"2024-09-15T02:43:09.455959+0000","modified":"2024-09-15T02:43:23.413315+0000","last_up_change":"2024-09-15T02:43:18.346896+0000","last_in_change":"2024-09-15T02:43:11.511047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:43:19.373532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:43:20.126006+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ab035ba6-bb97-46de-9069-4082e1295127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":33156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":33156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":33156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":33156}]},"public_addr":"172.21.15.53:6801/33156","cluster_addr":"172.21.15.53:6804/33156","heartbeat_back_addr":"172.21.15.53:6809/33156","heartbeat_front_addr":"172.21.15.53:6806/33156","state":["exists","up"]},{"osd":1,"uuid":"eb614e79-5e04-45a4-9962-84c7619ddb87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":33157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":33157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":33157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":33157}]},"public_addr":"172.21.15.53:6800/33157","cluster_addr":"172.21.15.53:6803/33157","heartbeat_back_addr":"172.21.15.53:6811/33157","heartbeat_front_addr":"172.21.15.53:6808/33157","state":["exists","up"]},{"osd":2,"uuid":"08ea49a0-28e1-4740-a4b4-d722560b15c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":33158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":33158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":33158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":33158}]},"public_addr":"172.21.15.53:6802/33158","cluster_addr":"172.21.15.53:6805/33158","heartbeat_back_addr":"172.21.15.53:6810/33158","heartbeat_front_addr":"172.21.15.53:6807/33158","state":["exists","up"]},{"osd":3,"uuid":"1a3a89c3-b449-46ba-bc65-3f53371d3af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":33159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":33159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6815","nonce":33159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":33159}]},"public_addr":"172.21.15.53:6812/33159","cluster_addr":"172.21.15.53:6813/33159","heartbeat_back_addr":"172.21.15.53:6815/33159","heartbeat_front_addr":"172.21.15.53:6814/33159","state":["exists","up"]},{"osd":4,"uuid":"d969fe78-c8c8-4c03-a534-2173406b0ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":33348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":33348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6815","nonce":33348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":33348}]},"public_addr":"172.21.15.104:6812/33348","cluster_addr":"172.21.15.104:6813/33348","heartbeat_back_addr":"172.21.15.104:6815/33348","heartbeat_front_addr":"172.21.15.104:6814/33348","state":["exists","up"]},{"osd":5,"uuid":"51e17a2f-42d3-49a6-b71e-dc7e13877c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":33346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":33346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":33346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":33346}]},"public_addr":"172.21.15.104:6800/33346","cluster_addr":"172.21.15.104:6801/33346","heartbeat_back_addr":"172.21.15.104:6803/33346","heartbeat_front_addr":"172.21.15.104:6802/33346","state":["exists","up"]},{"osd":6,"uuid":"c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":33357}]},"public_addr":"172.21.15.104:6808/33357","cluster_addr":"172.21.15.104:6809/33357","heartbeat_back_addr":"172.21.15.104:6811/33357","heartbeat_front_addr":"172.21.15.104:6810/33357","state":["exists","up"]},{"osd":7,"uuid":"7029aab4-31cc-4219-b7a4-a185564953b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":33347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":33347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":33347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":33347}]},"public_addr":"172.21.15.104:6804/33347","cluster_addr":"172.21.15.104:6805/33347","heartbeat_back_addr":"172.21.15.104:6807/33347","heartbeat_front_addr":"172.21.15.104:6806/33347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.615102+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.701424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.692098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.724924+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.688475+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.598581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.722216+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.661189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:43:24.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 10 librados: watch_flush enter 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 10 librados: watch_flush exit 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 20 client.4284.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 10 client.4284.objecter successfully canceled tick 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 >> v2:172.21.15.104:6816/32804 conn(0x7f4d1006e880 msgr2=0x7f4d10070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 --2- 172.21.15.53:0/1082349647 >> v2:172.21.15.104:6816/32804 conn(0x7f4d1006e880 0x7f4d10070d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c118ec0 tx=0x7f4d1c007e30 comp rx=0 tx=0).stop 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 10 monclient: shutdown 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d2c06bae0 msgr2=0x7f4d2c13dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d2c06bae0 0x7f4d2c13dc80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800b520 tx=0x7f4d280090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 shutdown_connections 2024-09-15T02:43:24.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d2c13a8e0 0x7f4d2c150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 --2- 172.21.15.53:0/1082349647 >> v2:172.21.15.104:6816/32804 conn(0x7f4d1006e880 0x7f4d10070d30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d2c06beb0 0x7f4d2c0a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 --2- 172.21.15.53:0/1082349647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d2c06bae0 0x7f4d2c13dc80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 >> 172.21.15.53:0/1082349647 conn(0x7f4d2c067b00 msgr2=0x7f4d2c0b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 shutdown_connections 2024-09-15T02:43:24.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 -- 172.21.15.53:0/1082349647 wait complete. 2024-09-15T02:43:24.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.051+0000 7f4d33b00640 1 librados: shutdown 2024-09-15T02:43:24.063 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T02:43:24.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:43:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.158+0000 7f9362d9d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.158+0000 7f9362d9d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.158+0000 7f9362d9d640 10 monclient: monmap: 2024-09-15T02:43:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.158955+0000 2024-09-15T02:43:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.158955+0000 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.160 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.158+0000 7f9362d9d640 1 Processor -- start 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.158+0000 7f9362d9d640 1 -- start start 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.158+0000 7f9362d9d640 10 monclient: init 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f935c06bae0 0x7f935c06b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f935c06bae0 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f935c06beb0 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f935c13a8e0 0x7f935c136f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f935c13a8e0 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c139d40 con 0x7f935c06beb0 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c06cda0 con 0x7f935c06bae0 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c06b6a0 con 0x7f935c13a8e0 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9362d9d640 10 monclient(hunting): authenticate will time out at 740.729650s 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f935c06bae0 0x7f935c06b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f935c06bae0 0x7f935c06b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51578/0 (socket says 172.21.15.53:51578) 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 1 -- 172.21.15.53:0/676800480 learned_addr learned my addr 172.21.15.53:0/676800480 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 10 monclient(hunting): get_auth_request con 0x7f935c06bae0 auth_method 0 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.159+0000 7f9360b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f936133a640 1 --2- 172.21.15.53:0/676800480 >> v2:172.21.15.104:3300/0 conn(0x7f935c13a8e0 0x7f935c136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f936133a640 10 monclient(hunting): get_auth_request con 0x7f935c13a8e0 auth_method 0 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f936133a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f936133a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f936133a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 1 -- 172.21.15.53:0/676800480 >> v2:172.21.15.104:3300/0 conn(0x7f935c13a8e0 msgr2=0x7f935c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 1 --2- 172.21.15.53:0/676800480 >> v2:172.21.15.104:3300/0 conn(0x7f935c13a8e0 0x7f935c136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 1 -- 172.21.15.53:0/676800480 >> v1:172.21.15.53:6789/0 conn(0x7f935c06beb0 legacy=0x7f935c13a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f935b7fe640 1 -- 172.21.15.53:0/676800480 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058132752 0 0) 0x7f935c139d40 con 0x7f935c06beb0 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 1 monclient: found mon.noname-a 2024-09-15T02:43:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 1 -- 172.21.15.53:0/676800480 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c06d080 con 0x7f935c06bae0 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f935b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f9360b39640 1 --2- 172.21.15.53:0/676800480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c06b160 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f935c06c740 tx=0x7f934c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924faae20f7b6cf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.160+0000 7f935b7fe640 1 -- 172.21.15.53:0/676800480 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c033550 con 0x7f935c06bae0 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: dump: 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 1 -- 172.21.15.53:0/676800480 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c0336b0 con 0x7f935c06bae0 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 5 monclient: authenticate success, global_id 4287 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:24.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 1 -- 172.21.15.53:0/676800480 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c033a50 con 0x7f935c06bae0 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 4 set_mon_vals no callback set 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f935b7fe640 10 monclient: dump: 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.174 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 10 monclient: shutdown 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 -- 172.21.15.53:0/676800480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 msgr2=0x7f935c06b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 --2- 172.21.15.53:0/676800480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c06b160 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f935c06c740 tx=0x7f934c00ba50 comp rx=0 tx=0).stop 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 -- 172.21.15.53:0/676800480 shutdown_connections 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 --2- 172.21.15.53:0/676800480 >> v2:172.21.15.104:3300/0 conn(0x7f935c13a8e0 0x7f935c136f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 --2- 172.21.15.53:0/676800480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c06b160 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 -- 172.21.15.53:0/676800480 >> 172.21.15.53:0/676800480 conn(0x7f935c067b00 msgr2=0x7f935c0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 -- 172.21.15.53:0/676800480 shutdown_connections 2024-09-15T02:43:24.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.161+0000 7f9362d9d640 1 -- 172.21.15.53:0/676800480 wait complete. 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 10 monclient: monmap: 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 1 librados: starting msgr at 2024-09-15T02:43:24.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 1 librados: starting objecter 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 1 Processor -- start 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 1 -- start start 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 1 librados: setting wanted keys 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 1 librados: calling monclient init 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.162+0000 7f9362d9d640 10 monclient: init 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c13dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f935c06bae0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f935c06beb0 0x7f935c1492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f935c06beb0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f935c13a8e0 0x7f935c1497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f935c13a8e0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c13cc00 con 0x7f935c06bae0 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c148dc0 con 0x7f935c13a8e0 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c13d740 con 0x7f935c06beb0 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 10 monclient(hunting): authenticate will time out at 740.733210s 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c13dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c13dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51588/0 (socket says 172.21.15.53:51588) 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 -- 172.21.15.53:0/3189737305 learned_addr learned my addr 172.21.15.53:0/3189737305 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): get_auth_request con 0x7f935c06bae0 auth_method 0 2024-09-15T02:43:24.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f935c06beb0 0x7f935c1492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 10 monclient(hunting): get_auth_request con 0x7f935c06beb0 auth_method 0 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f935c13a8e0 0x7f935c1497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 10 monclient(hunting): get_auth_request con 0x7f935c13a8e0 auth_method 0 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f935bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f936133a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 -- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f935c06beb0 msgr2=0x7f935c1492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f935c06beb0 0x7f935c1492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 -- 172.21.15.53:0/3189737305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f935c13a8e0 msgr2=0x7f935c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f935c13a8e0 0x7f935c1497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 monclient: found mon.noname-a 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 -- 172.21.15.53:0/3189737305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c00bcf0 con 0x7f935c06bae0 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9360b39640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c13dca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f934c00d1d0 tx=0x7f934c03b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 10 client.?.objecter ms_handle_connect 0x7f935c06bae0 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 1 -- 172.21.15.53:0/3189737305 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c035820 con 0x7f935c06bae0 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 10 monclient: dump: 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.181 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9362d9d640 5 monclient: authenticate success, global_id 4293 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.163+0000 7f9358ff9640 1 -- 172.21.15.53:0/3189737305 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c042070 con 0x7f935c06bae0 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9358ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9358ff9640 1 -- 172.21.15.53:0/3189737305 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c035bb0 con 0x7f935c06bae0 2024-09-15T02:43:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 monclient: _renew_subs 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c0a5e20 con 0x7f935c06bae0 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f935affd640 4 set_mon_vals no callback set 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9358ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9358ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9358ff9640 10 monclient: dump: 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 monclient: _renew_subs 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c13b1b0 con 0x7f935c06bae0 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 1 librados: init done 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.164+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c148f80 con 0x7f935c06bae0 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.165+0000 7f9358ff9640 1 -- 172.21.15.53:0/3189737305 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f934c003020 con 0x7f935c06bae0 2024-09-15T02:43:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.165+0000 7f9358ff9640 1 --2- 172.21.15.53:0/3189737305 >> v2:172.21.15.104:6816/32804 conn(0x7f933c06e8b0 0x7f933c070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f9358ff9640 1 -- 172.21.15.53:0/3189737305 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f934c0ae960 con 0x7f935c06bae0 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f9358ff9640 10 client.4293.objecter ms_dispatch 0x7f935c000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f9358ff9640 3 client.4293.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f9358ff9640 3 client.4293.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f9358ff9640 20 client.4293.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f935bfff640 1 --2- 172.21.15.53:0/3189737305 >> v2:172.21.15.104:6816/32804 conn(0x7f933c06e8b0 0x7f933c070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f935bfff640 10 monclient: get_auth_request con 0x7f933c06e8b0 auth_method 0 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f9358ff9640 10 client.4293.objecter ms_handle_connect 0x7f933c06e8b0 2024-09-15T02:43:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.166+0000 7f935bfff640 1 --2- 172.21.15.53:0/3189737305 >> v2:172.21.15.104:6816/32804 conn(0x7f933c06e8b0 0x7f933c070d60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f935c0a6700 tx=0x7f9344007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.169+0000 7f9358ff9640 1 -- 172.21.15.53:0/3189737305 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f934c078fa0 con 0x7f935c06bae0 2024-09-15T02:43:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.169+0000 7f9358ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.169+0000 7f9358ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:24.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.333+0000 7f9362d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:24.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.333+0000 7f9362d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:24.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.333+0000 7f9362d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.333+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f935c0646e0 con 0x7f935c06bae0 2024-09-15T02:43:24.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.334+0000 7f9358ff9640 1 -- 172.21.15.53:0/3189737305 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+11979 (secure 0 0 0) 0x7f934c07c7b0 con 0x7f935c06bae0 2024-09-15T02:43:24.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.334+0000 7f9358ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:43:24.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.334+0000 7f9358ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:24.335 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:43:24.335 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":18,"fsid":"7e7cb607-2b58-4a36-a082-99425792b70f","created":"2024-09-15T02:43:09.455959+0000","modified":"2024-09-15T02:43:23.413315+0000","last_up_change":"2024-09-15T02:43:18.346896+0000","last_in_change":"2024-09-15T02:43:11.511047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:43:19.373532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:43:20.126006+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ab035ba6-bb97-46de-9069-4082e1295127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":33156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":33156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":33156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":33156}]},"public_addr":"172.21.15.53:6801/33156","cluster_addr":"172.21.15.53:6804/33156","heartbeat_back_addr":"172.21.15.53:6809/33156","heartbeat_front_addr":"172.21.15.53:6806/33156","state":["exists","up"]},{"osd":1,"uuid":"eb614e79-5e04-45a4-9962-84c7619ddb87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":33157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":33157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":33157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":33157}]},"public_addr":"172.21.15.53:6800/33157","cluster_addr":"172.21.15.53:6803/33157","heartbeat_back_addr":"172.21.15.53:6811/33157","heartbeat_front_addr":"172.21.15.53:6808/33157","state":["exists","up"]},{"osd":2,"uuid":"08ea49a0-28e1-4740-a4b4-d722560b15c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":33158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":33158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":33158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":33158}]},"public_addr":"172.21.15.53:6802/33158","cluster_addr":"172.21.15.53:6805/33158","heartbeat_back_addr":"172.21.15.53:6810/33158","heartbeat_front_addr":"172.21.15.53:6807/33158","state":["exists","up"]},{"osd":3,"uuid":"1a3a89c3-b449-46ba-bc65-3f53371d3af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":33159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":33159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6815","nonce":33159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":33159}]},"public_addr":"172.21.15.53:6812/33159","cluster_addr":"172.21.15.53:6813/33159","heartbeat_back_addr":"172.21.15.53:6815/33159","heartbeat_front_addr":"172.21.15.53:6814/33159","state":["exists","up"]},{"osd":4,"uuid":"d969fe78-c8c8-4c03-a534-2173406b0ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":33348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":33348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6815","nonce":33348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":33348}]},"public_addr":"172.21.15.104:6812/33348","cluster_addr":"172.21.15.104:6813/33348","heartbeat_back_addr":"172.21.15.104:6815/33348","heartbeat_front_addr":"172.21.15.104:6814/33348","state":["exists","up"]},{"osd":5,"uuid":"51e17a2f-42d3-49a6-b71e-dc7e13877c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":33346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":33346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":33346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":33346}]},"public_addr":"172.21.15.104:6800/33346","cluster_addr":"172.21.15.104:6801/33346","heartbeat_back_addr":"172.21.15.104:6803/33346","heartbeat_front_addr":"172.21.15.104:6802/33346","state":["exists","up"]},{"osd":6,"uuid":"c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":33357}]},"public_addr":"172.21.15.104:6808/33357","cluster_addr":"172.21.15.104:6809/33357","heartbeat_back_addr":"172.21.15.104:6811/33357","heartbeat_front_addr":"172.21.15.104:6810/33357","state":["exists","up"]},{"osd":7,"uuid":"7029aab4-31cc-4219-b7a4-a185564953b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":33347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":33347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":33347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":33347}]},"public_addr":"172.21.15.104:6804/33347","cluster_addr":"172.21.15.104:6805/33347","heartbeat_back_addr":"172.21.15.104:6807/33347","heartbeat_front_addr":"172.21.15.104:6806/33347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.615102+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.701424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.692098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.724924+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.688475+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.598581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.722216+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.661189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:43:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 10 librados: watch_flush enter 2024-09-15T02:43:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 10 librados: watch_flush exit 2024-09-15T02:43:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 20 client.4293.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 10 client.4293.objecter successfully canceled tick 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 >> v2:172.21.15.104:6816/32804 conn(0x7f933c06e8b0 msgr2=0x7f933c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 1 --2- 172.21.15.53:0/3189737305 >> v2:172.21.15.104:6816/32804 conn(0x7f933c06e8b0 0x7f933c070d60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f935c0a6700 tx=0x7f9344007ca0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 10 monclient: shutdown 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 msgr2=0x7f935c13dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.335+0000 7f9362d9d640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c13dca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f934c00d1d0 tx=0x7f934c03b000 comp rx=0 tx=0).stop 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 shutdown_connections 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 --2- 172.21.15.53:0/3189737305 >> v2:172.21.15.104:6816/32804 conn(0x7f933c06e8b0 0x7f933c070d60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f935c13a8e0 0x7f935c1497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f935c06beb0 0x7f935c1492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 --2- 172.21.15.53:0/3189737305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f935c06bae0 0x7f935c13dca0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 >> 172.21.15.53:0/3189737305 conn(0x7f935c067b00 msgr2=0x7f935c06d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 shutdown_connections 2024-09-15T02:43:24.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 -- 172.21.15.53:0/3189737305 wait complete. 2024-09-15T02:43:24.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.336+0000 7f9362d9d640 1 librados: shutdown 2024-09-15T02:43:24.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T02:43:24.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T02:43:24.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T02:43:24.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T02:43:24.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T02:43:24.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T02:43:24.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-15T02:43:24.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.517+0000 7f335639d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.517+0000 7f335639d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.517+0000 7f335639d640 10 monclient: monmap: 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.518720+0000 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.518720+0000 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.519 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.518+0000 7f3b7d64f640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.518+0000 7f3b7d64f640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.518+0000 7f3b7d64f640 10 monclient: monmap: 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.519702+0000 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.519702+0000 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.521 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.519+0000 7f335639d640 1 Processor -- start 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.519+0000 7f335639d640 1 -- start start 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.519+0000 7f335639d640 10 monclient: init 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.519+0000 7fdad399d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.519+0000 7fdad399d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.519+0000 7fdad399d640 10 monclient: monmap: 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.522 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.520678+0000 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.520678+0000 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3348007320 0x7f3348035710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient(hunting): picked mon.noname-c con 0x7f3348007320 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348034570 0x7f3348030740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient(hunting): picked mon.noname-b con 0x7f3348034570 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3348037df0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3348034c90 con 0x7f3348037df0 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3348036760 con 0x7f3348007320 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3348037060 con 0x7f3348034570 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335639d640 10 monclient(hunting): authenticate will time out at 741.090350s 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7fa029edd640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7fa029edd640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f1168908640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f1168908640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f1168908640 10 monclient: monmap: 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.521384+0000 2024-09-15T02:43:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.521384+0000 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335493a640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3348037df0 0x7f3348038270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.53:50594/0 (socket says 172.21.15.53:50594) 2024-09-15T02:43:24.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f335493a640 1 -- 172.21.15.53:0/4206394018 learned_addr learned my addr 172.21.15.53:0/4206394018 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:24.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f3b7d64f640 1 Processor -- start 2024-09-15T02:43:24.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f334f7fe640 1 --2- 172.21.15.53:0/4206394018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348034570 0x7f3348030740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f3b7d64f640 1 -- start start 2024-09-15T02:43:24.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f3b7d64f640 10 monclient: init 2024-09-15T02:43:24.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334f7fe640 10 monclient(hunting): get_auth_request con 0x7f3348034570 auth_method 0 2024-09-15T02:43:24.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.519+0000 7f1300eef640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f1300eef640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7f1300eef640 10 monclient: monmap: 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.521289+0000 2024-09-15T02:43:24.529 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.521289+0000 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.520+0000 7fa029edd640 10 monclient: monmap: 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.521276+0000 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.521276+0000 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.531 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 1 --2- 172.21.15.53:0/4206394018 >> v2:172.21.15.104:3300/0 conn(0x7f3348007320 0x7f3348035710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): get_auth_request con 0x7f3348007320 auth_method 0 2024-09-15T02:43:24.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f1300eef640 1 Processor -- start 2024-09-15T02:43:24.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f1300eef640 1 -- start start 2024-09-15T02:43:24.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f1300eef640 10 monclient: init 2024-09-15T02:43:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7fa029edd640 1 Processor -- start 2024-09-15T02:43:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7fa029edd640 1 -- start start 2024-09-15T02:43:24.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7fa029edd640 10 monclient: init 2024-09-15T02:43:24.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334effd640 1 -- 172.21.15.53:0/4206394018 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325779423 0 0) 0x7f3348034c90 con 0x7f3348037df0 2024-09-15T02:43:24.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f3b7d64f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f3b7d64f640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f3b7d64f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b780af110 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f3b7d64f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3b78068b30 0x7f3b78140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f3b7d64f640 10 monclient(hunting): picked mon.noname-a con 0x7f3b78068b30 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f3b7d64f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3b78068f50 0x7f3b78141520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f3b7d64f640 10 monclient(hunting): picked mon.noname-c con 0x7f3b78068f50 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b7d64f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b780aea00 con 0x7f3b780af110 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b7d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b7d64f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b7806ee50 con 0x7f3b78068b30 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b7d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b7d64f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b78068630 con 0x7f3b78068f50 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b7d64f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b7d64f640 10 monclient(hunting): authenticate will time out at 741.092032s 2024-09-15T02:43:24.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3b78068f50 0x7f3b78141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3b78068f50 0x7f3b78141520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60170/0 (socket says 172.21.15.53:60170) 2024-09-15T02:43:24.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 -- 172.21.15.53:0/1809546605 learned_addr learned my addr 172.21.15.53:0/1809546605 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): get_auth_request con 0x7f3b78068f50 auth_method 0 2024-09-15T02:43:24.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334effd640 10 monclient(hunting): my global_id is 4285 2024-09-15T02:43:24.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f0ff6ce8640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f0ff6ce8640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f0ff6ce8640 10 monclient: monmap: 2024-09-15T02:43:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.521913+0000 2024-09-15T02:43:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.521913+0000 2024-09-15T02:43:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.539 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.539 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.539 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.539 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.539 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.540 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.540 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334effd640 1 -- 172.21.15.53:0/4206394018 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33400034e0 con 0x7f3348037df0 2024-09-15T02:43:24.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7f334ffff640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-09-15T02:43:24.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 1 -- 172.21.15.53:0/4206394018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348034570 msgr2=0x7f3348030740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 1 --2- 172.21.15.53:0/4206394018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348034570 0x7f3348030740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 1 -- 172.21.15.53:0/4206394018 >> v1:172.21.15.104:6789/0 conn(0x7f3348037df0 legacy=0x7f3348038270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 1 monclient: found mon.noname-c 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b767fc640 1 --2- 172.21.15.53:0/1809546605 >> v2:172.21.15.53:3300/0 conn(0x7f3b78068b30 0x7f3b78140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b767fc640 10 monclient(hunting): get_auth_request con 0x7f3b78068b30 auth_method 0 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2024-09-15T02:43:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 -- 172.21.15.53:0/1809546605 >> v2:172.21.15.53:3300/0 conn(0x7f3b78068b30 msgr2=0x7f3b78140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 --2- 172.21.15.53:0/1809546605 >> v2:172.21.15.53:3300/0 conn(0x7f3b78068b30 0x7f3b78140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 -- 172.21.15.53:0/1809546605 >> v1:172.21.15.53:6789/0 conn(0x7f3b780af110 legacy=0x7f3b7806cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 monclient: found mon.noname-c 2024-09-15T02:43:24.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b777fe640 1 -- 172.21.15.53:0/1809546605 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b78069a10 con 0x7f3b78068f50 2024-09-15T02:43:24.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f334ffff640 1 -- 172.21.15.53:0/4206394018 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3348036960 con 0x7f3348007320 2024-09-15T02:43:24.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f3b767fc640 1 --2- 172.21.15.53:0/1809546605 >> v2:172.21.15.53:3300/0 conn(0x7f3b78068b30 0x7f3b78140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:43:24.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f1168908640 1 Processor -- start 2024-09-15T02:43:24.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f1168908640 1 -- start start 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7f1168908640 10 monclient: init 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02406a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient(hunting): picked mon.noname-b con 0x7fa024145490 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa024069d30 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa02413eb10 0x7fa02413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient(hunting): picked mon.noname-a con 0x7fa02413eb10 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa024145860 con 0x7fa024069d30 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02406b330 con 0x7fa02413eb10 2024-09-15T02:43:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.521+0000 7fdad399d640 1 Processor -- start 2024-09-15T02:43:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7fdad399d640 1 -- start start 2024-09-15T02:43:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.522+0000 7fdad399d640 10 monclient: init 2024-09-15T02:43:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02406bdd0 con 0x7fa024145490 2024-09-15T02:43:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa029edd640 10 monclient(hunting): authenticate will time out at 741.093131s 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f0ff6ce8640 1 Processor -- start 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa0237fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02406a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa02413eb10 0x7fa02413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa0237fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02406a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:46990/0 (socket says 172.21.15.53:46990) 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa022ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa024069d30 0x7fa02406ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40724/0 (socket says 172.21.15.53:40724) 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 10 monclient(hunting): get_auth_request con 0x7fa02413eb10 auth_method 0 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa0237fe640 1 -- 172.21.15.53:0/3956827707 learned_addr learned my addr 172.21.15.53:0/3956827707 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa023fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa0237fe640 10 monclient(hunting): get_auth_request con 0x7fa024145490 auth_method 0 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa0237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa0237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fa0237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdacc0adcd0 0x7fdacc145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc0adcd0 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdacc0ae0a0 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc145df0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc06abd0 con 0x7fdacc0ae0a0 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdacc06a9d0 con 0x7fdacc0adcd0 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc1458d0 con 0x7fdacc145df0 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7fdad399d640 10 monclient(hunting): authenticate will time out at 741.093717s 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1164067280 0x7f116406aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient(hunting): picked mon.noname-a con 0x7f1164067280 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f116406af90 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11641416a0 0x7f1164141d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient(hunting): picked mon.noname-b con 0x7f11641416a0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fa0237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fa0237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fa0237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fa023fff640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdacc0adcd0 0x7fdacc145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc06d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47022/0 (socket says 172.21.15.53:47022) 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 1 -- 172.21.15.53:0/1091744221 learned_addr learned my addr 172.21.15.53:0/1091744221 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 10 monclient(hunting): get_auth_request con 0x7fdacc145df0 auth_method 0 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): get_auth_request con 0x7fdacc0adcd0 auth_method 0 2024-09-15T02:43:24.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f3b777fe640 1 --2- 172.21.15.53:0/1809546605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068f50 0x7f3b78141520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c009910 tx=0x7f3b6c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea3665d989db7fc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f334f7fe640 1 --2- 172.21.15.53:0/4206394018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348034570 0x7f3348030740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:24.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116406c680 con 0x7f116406af90 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116406b440 con 0x7f1164067280 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1164067650 con 0x7f11641416a0 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1168908640 10 monclient(hunting): authenticate will time out at 741.093716s 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f1162d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1164067280 0x7f116406aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1164067280 0x7f116406aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51616/0 (socket says 172.21.15.53:51616) 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 1 -- 172.21.15.53:0/4156101159 learned_addr learned my addr 172.21.15.53:0/4156101159 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): get_auth_request con 0x7f1164067280 auth_method 0 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-09-15T02:43:24.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 1 -- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11641416a0 msgr2=0x7f1164141d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fa023fff640 1 -- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 msgr2=0x7fa02406a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fa023fff640 1 --2- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02406a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T02:43:24.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fa023fff640 1 -- 172.21.15.53:0/3956827707 >> v1:172.21.15.53:6789/0 conn(0x7fa024069d30 legacy=0x7fa02406ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f12fc145440 0x7f12fc0aab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient(hunting): picked mon.noname-a con 0x7f12fc145440 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12fc145f00 0x7f12fc06b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient(hunting): picked mon.noname-c con 0x7f12fc145f00 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12fc06a8c0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12fc145810 con 0x7f12fc06a8c0 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12fc06ade0 con 0x7f12fc145440 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f116359e640 1 --2- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11641416a0 0x7f1164141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 1 --2- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11641416a0 0x7f1164141d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 1 -- 172.21.15.53:0/4156101159 >> v1:172.21.15.104:6789/0 conn(0x7f116406af90 legacy=0x7f1164140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f0ff6ce8640 1 -- start start 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.523+0000 7f0ff6ce8640 10 monclient: init 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12fc0aa7b0 con 0x7f12fc145f00 2024-09-15T02:43:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1300eef640 10 monclient(hunting): authenticate will time out at 741.094611s 2024-09-15T02:43:24.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T02:43:24.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ff013cc80 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:24.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 1 monclient: found mon.noname-a 2024-09-15T02:43:24.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f1162d9d640 1 -- 172.21.15.53:0/4156101159 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1164142290 con 0x7f1164067280 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 1 -- 172.21.15.53:0/1091744221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 msgr2=0x7fdacc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 1 --2- 172.21.15.53:0/1091744221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 1 -- 172.21.15.53:0/1091744221 >> v1:172.21.15.53:6789/0 conn(0x7fdacc0ae0a0 legacy=0x7fdacc141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 1 monclient: found mon.noname-a 2024-09-15T02:43:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7fdad1739640 1 -- 172.21.15.53:0/1091744221 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc0af360 con 0x7fdacc0adcd0 2024-09-15T02:43:24.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0ff013d930 0x7f0ff006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff013d930 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d050 0x7f0ff006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff013d050 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff4a84640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0ff013cc80 0x7f0ff006e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.53:50610/0 (socket says 172.21.15.53:50610) 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff4a84640 1 -- 172.21.15.53:0/2755006237 learned_addr learned my addr 172.21.15.53:0/2755006237 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff0137360 con 0x7f0ff013cc80 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff006a640 con 0x7f0ff013d930 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f116359e640 10 monclient: get_auth_request con 0x7f11641416a0 auth_method 0 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f116359e640 1 --2- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11641416a0 0x7f1164141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:43:24.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff006c310 con 0x7f0ff013d050 2024-09-15T02:43:24.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.524+0000 7f0ff6ce8640 10 monclient(hunting): authenticate will time out at 741.094994s 2024-09-15T02:43:24.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f334ffff640 1 --2- 172.21.15.53:0/4206394018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348007320 0x7f3348035710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f333c009850 tx=0x7f333c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f560e83b84dd6188 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 1 --2- 172.21.15.53:0/2755006237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d050 0x7f0ff006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0fef7fe640 1 -- 172.21.15.53:0/2755006237 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388466684 0 0) 0x7f0ff0137360 con 0x7f0ff013cc80 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0fef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0fef7fe640 10 monclient(hunting): my global_id is 4294 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0fef7fe640 1 -- 172.21.15.53:0/2755006237 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe00034e0 con 0x7f0ff013cc80 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): get_auth_request con 0x7f0ff013d050 auth_method 0 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 1 --2- 172.21.15.53:0/2755006237 >> v2:172.21.15.53:3300/0 conn(0x7f0ff013d930 0x7f0ff006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 10 monclient(hunting): get_auth_request con 0x7f0ff013d930 auth_method 0 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0feffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-09-15T02:43:24.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 1 -- 172.21.15.53:0/2755006237 >> v2:172.21.15.53:3300/0 conn(0x7f0ff013d930 msgr2=0x7f0ff006ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 1 --2- 172.21.15.53:0/2755006237 >> v2:172.21.15.53:3300/0 conn(0x7f0ff013d930 0x7f0ff006ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 1 -- 172.21.15.53:0/2755006237 >> v1:172.21.15.104:6789/0 conn(0x7f0ff013cc80 legacy=0x7f0ff006e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 1 monclient: found mon.noname-b 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f0ff5285640 1 -- 172.21.15.53:0/2755006237 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff006c580 con 0x7f0ff013d050 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f12fc145440 0x7f12fc0aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f12fc145440 0x7f12fc0aab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51640/0 (socket says 172.21.15.53:51640) 2024-09-15T02:43:24.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7fa023fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 1 -- 172.21.15.53:0/11644513 learned_addr learned my addr 172.21.15.53:0/11644513 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 10 monclient(hunting): get_auth_request con 0x7f12fc145440 auth_method 0 2024-09-15T02:43:24.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12f9d9b640 1 --2- 172.21.15.53:0/11644513 >> v2:172.21.15.104:3300/0 conn(0x7f12fc145f00 0x7f12fc06b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12f9d9b640 10 monclient(hunting): get_auth_request con 0x7f12fc145f00 auth_method 0 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f12f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-09-15T02:43:24.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 1 -- 172.21.15.53:0/11644513 >> v2:172.21.15.104:3300/0 conn(0x7f12fc145f00 msgr2=0x7f12fc06b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 1 --2- 172.21.15.53:0/11644513 >> v2:172.21.15.104:3300/0 conn(0x7f12fc145f00 0x7f12fc06b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 1 -- 172.21.15.53:0/11644513 >> v1:172.21.15.104:6789/0 conn(0x7f12fc06a8c0 legacy=0x7f12fc13ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 1 monclient: found mon.noname-a 2024-09-15T02:43:24.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7fa023fff640 1 monclient: found mon.noname-a 2024-09-15T02:43:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 1 -- 172.21.15.53:0/4206394018 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335004c4a0 con 0x7f3348007320 2024-09-15T02:43:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 10 monclient: dump: 2024-09-15T02:43:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f335639d640 5 monclient: authenticate success, global_id 4288 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f335639d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 1 -- 172.21.15.53:0/4206394018 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c02a870 con 0x7f3348007320 2024-09-15T02:43:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f334effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7fa023fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7fdad1739640 1 --2- 172.21.15.53:0/1091744221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 0x7fdacc145200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009850 tx=0x7fdac80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bbe08b1a39c1d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7fa023fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7fa023fff640 1 -- 172.21.15.53:0/3956827707 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02406c0b0 con 0x7fa02413eb10 2024-09-15T02:43:24.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7fa0237fe640 1 --2- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02406a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:24.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7fa023fff640 1 --2- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 0x7fa02413ef90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009e60 tx=0x7fa01c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aba990916423172 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 1 -- 172.21.15.53:0/1809546605 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c02ad90 con 0x7f3b78068f50 2024-09-15T02:43:24.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f1162d9d640 1 --2- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1164067280 0x7f116406aa50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1154009910 tx=0x7f11540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7883b175b9e271f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 10 monclient: dump: 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.573 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.574 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f335639d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f335639d640 4 set_mon_vals no callback set 2024-09-15T02:43:24.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b7d64f640 5 monclient: authenticate success, global_id 4291 2024-09-15T02:43:24.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f0ff5285640 1 --2- 172.21.15.53:0/2755006237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d050 0x7f0ff006b2b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009e60 tx=0x7f0fe8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27826ce5a96d96aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b7d64f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:24.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f334effd640 1 -- 172.21.15.53:0/4206394018 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c02ac50 con 0x7f3348007320 2024-09-15T02:43:24.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f334effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 1 -- 172.21.15.53:0/4156101159 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115402adb0 con 0x7f1164067280 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: dump: 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.576 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f334effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f334effd640 10 monclient: dump: 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.577 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 1 -- 172.21.15.53:0/1809546605 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c032440 con 0x7f3b78068f50 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b75ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b7d64f640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.525+0000 7f3b7d64f640 4 set_mon_vals no callback set 2024-09-15T02:43:24.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 1 -- 172.21.15.53:0/4156101159 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1154032440 con 0x7f1164067280 2024-09-15T02:43:24.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 10 monclient: shutdown 2024-09-15T02:43:24.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 1 -- 172.21.15.53:0/1809546605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068f50 msgr2=0x7f3b78141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 1 --2- 172.21.15.53:0/1809546605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068f50 0x7f3b78141520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c009910 tx=0x7f3b6c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 1 -- 172.21.15.53:0/1809546605 shutdown_connections 2024-09-15T02:43:24.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 1 --2- 172.21.15.53:0/1809546605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068f50 0x7f3b78141520 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 1 --2- 172.21.15.53:0/1809546605 >> v2:172.21.15.53:3300/0 conn(0x7f3b78068b30 0x7f3b78140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f3b7d64f640 1 -- 172.21.15.53:0/1809546605 >> 172.21.15.53:0/1809546605 conn(0x7f3b78062d90 msgr2=0x7f3b78063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 1 -- 172.21.15.53:0/4156101159 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115402adb0 con 0x7f1164067280 2024-09-15T02:43:24.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1161d9b640 10 monclient: dump: 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1168908640 5 monclient: authenticate success, global_id 4302 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1168908640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1168908640 4 set_mon_vals no callback set 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f3b7d64f640 1 -- 172.21.15.53:0/1809546605 shutdown_connections 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 1 -- 172.21.15.53:0/3956827707 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c007550 con 0x7fa02413eb10 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: dump: 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.582 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 1 -- 172.21.15.53:0/3956827707 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01c02d050 con 0x7fa02413eb10 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 1 -- 172.21.15.53:0/3956827707 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c007920 con 0x7fa02413eb10 2024-09-15T02:43:24.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 10 monclient: shutdown 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 -- 172.21.15.53:0/4206394018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348007320 msgr2=0x7f3348035710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 --2- 172.21.15.53:0/4206394018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348007320 0x7f3348035710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f333c009850 tx=0x7f333c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 -- 172.21.15.53:0/4206394018 shutdown_connections 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 --2- 172.21.15.53:0/4206394018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348034570 0x7f3348030740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 --2- 172.21.15.53:0/4206394018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348007320 0x7f3348035710 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 -- 172.21.15.53:0/4206394018 >> 172.21.15.53:0/4206394018 conn(0x7f3348039ef0 msgr2=0x7f334803a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 1 -- 172.21.15.53:0/1091744221 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac802ad00 con 0x7fdacc0adcd0 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: dump: 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 1 -- 172.21.15.53:0/1091744221 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac802ae60 con 0x7fdacc0adcd0 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 1 -- 172.21.15.53:0/1091744221 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8032670 con 0x7fdacc0adcd0 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdac3fff640 10 monclient: dump: 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.586 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa0227fc640 10 monclient: dump: 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.587 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.526+0000 7f12fa59c640 1 -- 172.21.15.53:0/11644513 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc06b840 con 0x7f12fc145440 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12fa59c640 1 --2- 172.21.15.53:0/11644513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc145440 0x7f12fc0aab50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12fc06a6c0 tx=0x7f12ec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c408049d44b105f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 1 -- 172.21.15.53:0/2755006237 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8007550 con 0x7f0ff013d050 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: dump: 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.589 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 -- 172.21.15.53:0/4206394018 shutdown_connections 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 1 -- 172.21.15.53:0/2755006237 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe802d050 con 0x7f0ff013d050 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0ff6ce8640 5 monclient: authenticate success, global_id 4262 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0ff6ce8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 1 -- 172.21.15.53:0/2755006237 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8007900 con 0x7f0ff013d050 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0fef7fe640 10 monclient: dump: 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0ff6ce8640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f0ff6ce8640 4 set_mon_vals no callback set 2024-09-15T02:43:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f335639d640 1 -- 172.21.15.53:0/4206394018 wait complete. 2024-09-15T02:43:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa029edd640 5 monclient: authenticate success, global_id 4299 2024-09-15T02:43:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa029edd640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fa029edd640 4 set_mon_vals no callback set 2024-09-15T02:43:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f3b7d64f640 1 -- 172.21.15.53:0/1809546605 wait complete. 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f1168908640 10 monclient: shutdown 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f1168908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f1168908640 1 -- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1164067280 msgr2=0x7f116406aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f1168908640 1 --2- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1164067280 0x7f116406aa50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1154009910 tx=0x7f11540091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 10 monclient: shutdown 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d050 msgr2=0x7f0ff006b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 1 --2- 172.21.15.53:0/2755006237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d050 0x7f0ff006b2b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009e60 tx=0x7f0fe8009780 comp rx=0 tx=0).stop 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f335639d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f335639d640 10 monclient: monmap: 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.594 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdad399d640 5 monclient: authenticate success, global_id 4305 2024-09-15T02:43:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdad399d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7fdad399d640 4 set_mon_vals no callback set 2024-09-15T02:43:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fdad399d640 10 monclient: shutdown 2024-09-15T02:43:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fdad399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fdad399d640 1 -- 172.21.15.53:0/1091744221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 msgr2=0x7fdacc145200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fdad399d640 1 --2- 172.21.15.53:0/1091744221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 0x7fdacc145200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009850 tx=0x7fdac80090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 10 monclient: shutdown 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 1 -- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 msgr2=0x7fa02413ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 1 --2- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 0x7fa02413ef90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009e60 tx=0x7fa01c009780 comp rx=0 tx=0).stop 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 1 -- 172.21.15.53:0/3956827707 shutdown_connections 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 1 --2- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 0x7fa02413ef90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 1 --2- 172.21.15.53:0/3956827707 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02406a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7fa029edd640 1 -- 172.21.15.53:0/3956827707 >> 172.21.15.53:0/3956827707 conn(0x7fa0240b4400 msgr2=0x7fa0240b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f3b7d64f640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f3b7d64f640 10 monclient: monmap: 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.597 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 shutdown_connections 2024-09-15T02:43:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 1 --2- 172.21.15.53:0/2755006237 >> v2:172.21.15.53:3300/0 conn(0x7f0ff013d930 0x7f0ff006ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 1 --2- 172.21.15.53:0/2755006237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d050 0x7f0ff006b2b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 >> 172.21.15.53:0/2755006237 conn(0x7f0ff0067280 msgr2=0x7f0ff0067680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f335639d640 1 librados: starting msgr at 2024-09-15T02:43:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f335639d640 1 librados: starting objecter 2024-09-15T02:43:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f905ee4d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f905ee4d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.528+0000 7f905ee4d640 10 monclient: monmap: 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:24.529408+0000 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:24.529408+0000 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fdad399d640 1 -- 172.21.15.53:0/1091744221 shutdown_connections 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fdad399d640 1 --2- 172.21.15.53:0/1091744221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fdad399d640 1 --2- 172.21.15.53:0/1091744221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 0x7fdacc145200 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fdad399d640 1 -- 172.21.15.53:0/1091744221 >> 172.21.15.53:0/1091744221 conn(0x7fdacc062d90 msgr2=0x7fdacc063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f3b7d64f640 1 librados: starting msgr at 2024-09-15T02:43:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f3b7d64f640 1 librados: starting objecter 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 1 -- 172.21.15.53:0/11644513 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec02ad70 con 0x7f12fc145440 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 10 monclient: dump: 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1300eef640 5 monclient: authenticate success, global_id 4311 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1300eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 1 -- 172.21.15.53:0/11644513 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec032440 con 0x7f12fc145440 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f12f959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1300eef640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.527+0000 7f1300eef640 4 set_mon_vals no callback set 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 10 monclient: shutdown 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 -- 172.21.15.53:0/11644513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc145440 msgr2=0x7f12fc0aab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f12f959a640 1 -- 172.21.15.53:0/11644513 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec02ad70 con 0x7f12fc145440 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 --2- 172.21.15.53:0/11644513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc145440 0x7f12fc0aab50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12fc06a6c0 tx=0x7f12ec0091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 shutdown_connections 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fa029edd640 1 -- 172.21.15.53:0/3956827707 shutdown_connections 2024-09-15T02:43:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fa029edd640 1 -- 172.21.15.53:0/3956827707 wait complete. 2024-09-15T02:43:24.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f335639d640 1 Processor -- start 2024-09-15T02:43:24.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/2755006237 wait complete. 2024-09-15T02:43:24.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1168908640 1 -- 172.21.15.53:0/4156101159 shutdown_connections 2024-09-15T02:43:24.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1168908640 1 --2- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11641416a0 0x7f1164141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1168908640 1 --2- 172.21.15.53:0/4156101159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1164067280 0x7f116406aa50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1168908640 1 -- 172.21.15.53:0/4156101159 >> 172.21.15.53:0/4156101159 conn(0x7f11640b0510 msgr2=0x7f11640b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f3b7d64f640 1 Processor -- start 2024-09-15T02:43:24.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fdad399d640 1 -- 172.21.15.53:0/1091744221 shutdown_connections 2024-09-15T02:43:24.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1168908640 1 -- 172.21.15.53:0/4156101159 shutdown_connections 2024-09-15T02:43:24.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1168908640 1 -- 172.21.15.53:0/4156101159 wait complete. 2024-09-15T02:43:24.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f12f959a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T02:43:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 -- 172.21.15.53:0/11644513 shutdown_connections 2024-09-15T02:43:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 --2- 172.21.15.53:0/11644513 >> v2:172.21.15.104:3300/0 conn(0x7f12fc145f00 0x7f12fc06b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 --2- 172.21.15.53:0/11644513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc145440 0x7f12fc0aab50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 -- 172.21.15.53:0/11644513 >> 172.21.15.53:0/11644513 conn(0x7f12fc067b60 msgr2=0x7f12fc0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f335639d640 1 -- start start 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f335639d640 1 librados: setting wanted keys 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f335639d640 1 librados: calling monclient init 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f335639d640 10 monclient: init 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fa029edd640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7fa029edd640 10 monclient: monmap: 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.609 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.610 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.610 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.610 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.610 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f905ee4d640 1 Processor -- start 2024-09-15T02:43:24.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 -- 172.21.15.53:0/11644513 shutdown_connections 2024-09-15T02:43:24.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f3b7d64f640 1 -- start start 2024-09-15T02:43:24.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f3b7d64f640 1 librados: setting wanted keys 2024-09-15T02:43:24.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f3b7d64f640 1 librados: calling monclient init 2024-09-15T02:43:24.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f3b7d64f640 10 monclient: init 2024-09-15T02:43:24.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7fa029edd640 1 librados: starting msgr at 2024-09-15T02:43:24.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7fa029edd640 1 librados: starting objecter 2024-09-15T02:43:24.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f905ee4d640 1 -- start start 2024-09-15T02:43:24.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f905ee4d640 10 monclient: init 2024-09-15T02:43:24.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f1168908640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f1168908640 10 monclient: monmap: 2024-09-15T02:43:24.612 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7fa029edd640 1 Processor -- start 2024-09-15T02:43:24.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f335639d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f335639d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:24.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7fa029edd640 1 -- start start 2024-09-15T02:43:24.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7fa029edd640 1 librados: setting wanted keys 2024-09-15T02:43:24.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7fa029edd640 1 librados: calling monclient init 2024-09-15T02:43:24.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7fa029edd640 10 monclient: init 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f335639d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348007320 0x7f33480dde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f1168908640 1 librados: starting msgr at 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f1168908640 1 librados: starting objecter 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f3b7d64f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f3b7d64f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f3b7d64f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068b30 0x7f3b780aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f3b7d64f640 10 monclient(hunting): picked mon.noname-b con 0x7f3b78068b30 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f3b7d64f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b78068f50 0x7f3b781428e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): picked mon.noname-c con 0x7f3348007320 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348034570 0x7f33480de390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T02:43:24.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): picked mon.noname-b con 0x7f3348034570 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 10 monclient(hunting): picked mon.noname-c con 0x7f3b78068f50 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b780af110 0x7f3b7814d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 10 monclient(hunting): picked mon.noname-a con 0x7f3b780af110 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b78141f30 con 0x7f3b780af110 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b780ab8a0 con 0x7f3b78068b30 2024-09-15T02:43:24.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7813ac40 con 0x7f3b78068f50 2024-09-15T02:43:24.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b7d64f640 10 monclient(hunting): authenticate will time out at 741.101254s 2024-09-15T02:43:24.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3348037df0 0x7f334802d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): picked mon.noname-a con 0x7f3348037df0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa024069d30 0x7fa02406e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): picked mon.noname-b con 0x7fa024069d30 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 0x7fa02406e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): picked mon.noname-a con 0x7fa02413eb10 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02413d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): picked mon.noname-c con 0x7fa024145490 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02406d930 con 0x7fa02413eb10 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02406c890 con 0x7fa024069d30 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02406ca50 con 0x7fa024145490 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa029edd640 10 monclient(hunting): authenticate will time out at 741.101707s 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): picked mon.noname-b con 0x7f90580ad160 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9058141cf0 0x7f9058142100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): picked mon.noname-c con 0x7f9058141cf0 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9058142640 0x7f90580aa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): picked mon.noname-a con 0x7f9058142640 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905806b570 con 0x7f9058142640 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90580ad530 con 0x7f9058141cf0 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90580ac730 con 0x7f90580ad160 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905ee4d640 10 monclient(hunting): authenticate will time out at 741.101592s 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905de4b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348034570 0x7f33480de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33480debf0 con 0x7f3348037df0 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334800ade0 con 0x7f3348034570 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348034570 0x7f33480de390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60188/0 (socket says 172.21.15.53:60188) 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33480de8d0 con 0x7f3348007320 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334f7fe640 1 -- 172.21.15.53:0/2636528188 learned_addr learned my addr 172.21.15.53:0/2636528188 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335639d640 10 monclient(hunting): authenticate will time out at 741.101405s 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3348037df0 0x7f334802d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 10 monclient(hunting): get_auth_request con 0x7f3348037df0 auth_method 0 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334ffff640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348007320 0x7f33480dde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f1168908640 1 Processor -- start 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f1168908640 1 -- start start 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f1168908640 1 librados: setting wanted keys 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f1168908640 1 librados: calling monclient init 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f1168908640 10 monclient: init 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b780af110 0x7f3b7814d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b780af110 0x7f3b7814d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51666/0 (socket says 172.21.15.53:51666) 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 1 -- 172.21.15.53:0/448214785 learned_addr learned my addr 172.21.15.53:0/448214785 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 10 monclient(hunting): get_auth_request con 0x7f3b780af110 auth_method 0 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 1 --2- 172.21.15.53:0/448214785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b78068f50 0x7f3b781428e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905d64a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9058141cf0 0x7f9058142100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905de4b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47074/0 (socket says 172.21.15.53:47074) 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905d64a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9058141cf0 0x7f9058142100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60192/0 (socket says 172.21.15.53:60192) 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905de4b640 1 -- 172.21.15.53:0/2148650200 learned_addr learned my addr 172.21.15.53:0/2148650200 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905de4b640 10 monclient(hunting): get_auth_request con 0x7f90580ad160 auth_method 0 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905de4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905e64c640 1 --2- 172.21.15.53:0/2148650200 >> v2:172.21.15.53:3300/0 conn(0x7f9058142640 0x7f90580aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905de4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905de4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905e64c640 10 monclient(hunting): get_auth_request con 0x7f9058142640 auth_method 0 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905e64c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905e64c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905e64c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334ffff640 10 monclient(hunting): get_auth_request con 0x7f3348007320 auth_method 0 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f335493a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 10 monclient(hunting): get_auth_request con 0x7f3b78068f50 auth_method 0 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905d64a640 10 monclient(hunting): get_auth_request con 0x7f9058141cf0 auth_method 0 2024-09-15T02:43:24.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f905d64a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f334ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7fa0237fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa024069d30 0x7fa02406e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa024069d30 0x7fa02406e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60206/0 (socket says 172.21.15.53:60206) 2024-09-15T02:43:24.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f1300eef640 1 -- 172.21.15.53:0/11644513 wait complete. 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 -- 172.21.15.53:0/849433396 learned_addr learned my addr 172.21.15.53:0/849433396 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): get_auth_request con 0x7fa024069d30 auth_method 0 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f1300eef640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f1300eef640 10 monclient: monmap: 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.629 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f3b76ffd640 1 --2- 172.21.15.53:0/448214785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068b30 0x7f3b780aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b76ffd640 10 monclient(hunting): get_auth_request con 0x7f3b78068b30 auth_method 0 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f1300eef640 1 librados: starting msgr at 2024-09-15T02:43:24.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.531+0000 7f1300eef640 1 librados: starting objecter 2024-09-15T02:43:24.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-15T02:43:24.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905e64c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905e64c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905e64c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 1 --2- 172.21.15.53:0/849433396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02413d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 1 --2- 172.21.15.53:0/849433396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 0x7fa02406e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1164067280 0x7f1164145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): picked mon.noname-c con 0x7f1164067280 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f116406af90 0x7f1164142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 1 -- 172.21.15.53:0/448214785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068b30 msgr2=0x7f3b780aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 1 --2- 172.21.15.53:0/448214785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b78068b30 0x7f3b780aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 1 -- 172.21.15.53:0/448214785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b780af110 msgr2=0x7f3b7814d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 1 --2- 172.21.15.53:0/448214785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b780af110 0x7f3b7814d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 1 monclient: found mon.noname-c 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 10 monclient(hunting): get_auth_request con 0x7fa02413eb10 auth_method 0 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 10 monclient(hunting): get_auth_request con 0x7fa024145490 auth_method 0 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 1 -- 172.21.15.53:0/2148650200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 msgr2=0x7f905806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): picked mon.noname-a con 0x7f116406af90 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11641416a0 0x7f11641434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): picked mon.noname-b con 0x7f11641416a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11640a5830 con 0x7f116406af90 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11640a6c50 con 0x7f11641416a0 2024-09-15T02:43:24.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11640a6f30 con 0x7f1164067280 2024-09-15T02:43:24.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f3b767fc640 1 -- 172.21.15.53:0/448214785 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c009450 con 0x7f3b78068f50 2024-09-15T02:43:24.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 1 --2- 172.21.15.53:0/2148650200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 1 -- 172.21.15.53:0/2148650200 >> v2:172.21.15.53:3300/0 conn(0x7f9058142640 msgr2=0x7f90580aa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:24.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1168908640 10 monclient(hunting): authenticate will time out at 741.102354s 2024-09-15T02:43:24.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1162d9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1164067280 0x7f1164145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 1 --2- 172.21.15.53:0/2148650200 >> v2:172.21.15.53:3300/0 conn(0x7f9058142640 0x7f90580aa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11641416a0 0x7f11641434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f116406af90 0x7f1164142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 1 monclient: found mon.noname-c 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1162d9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1164067280 0x7f1164145a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47094/0 (socket says 172.21.15.53:47094) 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1162d9d640 1 -- 172.21.15.53:0/3127262760 learned_addr learned my addr 172.21.15.53:0/3127262760 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11641416a0 0x7f11641434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60216/0 (socket says 172.21.15.53:60216) 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa023fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116259c640 10 monclient(hunting): get_auth_request con 0x7f116406af90 auth_method 0 2024-09-15T02:43:24.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa022ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 -- 172.21.15.53:0/849433396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 msgr2=0x7fa02413d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 --2- 172.21.15.53:0/849433396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa024145490 0x7fa02413d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 -- 172.21.15.53:0/849433396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 msgr2=0x7fa02406e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 --2- 172.21.15.53:0/849433396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02413eb10 0x7fa02406e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 monclient: found mon.noname-b 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905d64a640 1 -- 172.21.15.53:0/2148650200 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905806b210 con 0x7f9058141cf0 2024-09-15T02:43:24.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fa0237fe640 1 -- 172.21.15.53:0/849433396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014009450 con 0x7fa024069d30 2024-09-15T02:43:24.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 10 monclient(hunting): get_auth_request con 0x7f11641416a0 auth_method 0 2024-09-15T02:43:24.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1162d9d640 10 monclient(hunting): get_auth_request con 0x7f1164067280 auth_method 0 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1162d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1162d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1162d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f905e64c640 1 --2- 172.21.15.53:0/2148650200 >> v2:172.21.15.53:3300/0 conn(0x7f9058142640 0x7f90580aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:24.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f1300eef640 1 Processor -- start 2024-09-15T02:43:24.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f116359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b767fc640 1 --2- 172.21.15.53:0/448214785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b78068f50 0x7f3b781428e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000d180 tx=0x7f3b700099d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f334ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f334f7fe640 10 monclient(hunting): get_auth_request con 0x7f3348034570 auth_method 0 2024-09-15T02:43:24.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f334f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f334f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7f334f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f335493a640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-09-15T02:43:24.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f335493a640 1 -- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348007320 msgr2=0x7f33480dde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 client.?.objecter ms_handle_connect 0x7f3b78068f50 2024-09-15T02:43:24.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.529+0000 7f0ff6ce8640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 10 monclient(hunting): handle_auth_done global_id 4309 payload 824 2024-09-15T02:43:24.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa0237fe640 1 --2- 172.21.15.53:0/849433396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa024069d30 0x7fa02406e030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa01400ea30 tx=0x7fa0140091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 1 -- 172.21.15.53:0/3127262760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1164067280 msgr2=0x7f1164145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 10 monclient: monmap: 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 1 librados: starting msgr at 2024-09-15T02:43:24.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 1 librados: starting objecter 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 1 Processor -- start 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 1 -- start start 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 1 librados: setting wanted keys 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 1 librados: calling monclient init 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.530+0000 7f0ff6ce8640 10 monclient: init 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70015440 con 0x7f3b78068f50 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:24.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: dump: 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ff013cc80 0x7f0ff0139b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff013cc80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ff013d050 0x7f0ff00a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff013d050 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d930 0x7f0ff016e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff013d930 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff00a74f0 con 0x7f0ff013cc80 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff00a7be0 con 0x7f0ff013d050 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff00a9d70 con 0x7f0ff013d930 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff6ce8640 10 monclient(hunting): authenticate will time out at 741.103564s 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d930 0x7f0ff016e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d930 0x7f0ff016e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47098/0 (socket says 172.21.15.53:47098) 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 1 -- 172.21.15.53:0/1769696487 learned_addr learned my addr 172.21.15.53:0/1769696487 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 10 monclient(hunting): get_auth_request con 0x7f0ff013d930 auth_method 0 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f0ff5285640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 1 --2- 172.21.15.53:0/3127262760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1164067280 0x7f1164145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 10 client.?.objecter ms_handle_connect 0x7fa024069d30 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01400ca10 con 0x7fa024069d30 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 10 monclient: dump: 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.654 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa00b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 5 monclient: authenticate success, global_id 4300 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 10 monclient: _renew_subs 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c009a20 con 0x7fa024069d30 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 10 monclient: _renew_subs 2024-09-15T02:43:24.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02413dd70 con 0x7fa024069d30 2024-09-15T02:43:24.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fa029edd640 1 librados: init done 2024-09-15T02:43:24.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905d64a640 1 --2- 172.21.15.53:0/2148650200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058141cf0 0x7f9058142100 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90580ad7d0 tx=0x7f90480090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a4044e9b9d868b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f1300eef640 1 -- start start 2024-09-15T02:43:24.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f1300eef640 1 librados: setting wanted keys 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f1300eef640 1 librados: calling monclient init 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f1300eef640 10 monclient: init 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f1300eef640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f1300eef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc06a8c0 0x7f12fc06e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): picked mon.noname-a con 0x7f12fc06a8c0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12fc145440 0x7f12fc159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): picked mon.noname-b con 0x7f12fc145440 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12fc145f00 0x7f12fc141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): picked mon.noname-c con 0x7f12fc145f00 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc1416b0 con 0x7f12fc06a8c0 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc06ed30 con 0x7f12fc145440 2024-09-15T02:43:24.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc15a810 con 0x7f12fc145f00 2024-09-15T02:43:24.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f1300eef640 10 monclient(hunting): authenticate will time out at 741.104080s 2024-09-15T02:43:24.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.532+0000 7fdad399d640 1 -- 172.21.15.53:0/1091744221 wait complete. 2024-09-15T02:43:24.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 10 monclient: monmap: 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 1 librados: starting msgr at 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 1 librados: starting objecter 2024-09-15T02:43:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 1 Processor -- start 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 1 -- start start 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 1 librados: setting wanted keys 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 1 librados: calling monclient init 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7fdad399d640 10 monclient: init 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 1 -- 172.21.15.53:0/2148650200 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904802ac70 con 0x7f9058141cf0 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: dump: 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 1 -- 172.21.15.53:0/2148650200 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904802add0 con 0x7f9058141cf0 2024-09-15T02:43:24.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12f9d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12fc145440 0x7f12fc159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12fa59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc06a8c0 0x7f12fc06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12fa59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc06a8c0 0x7f12fc06e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51708/0 (socket says 172.21.15.53:51708) 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12fa59c640 1 -- 172.21.15.53:0/2521790191 learned_addr learned my addr 172.21.15.53:0/2521790191 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12fa59c640 10 monclient(hunting): get_auth_request con 0x7f12fc06a8c0 auth_method 0 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12f9d9b640 10 monclient(hunting): get_auth_request con 0x7f12fc145440 auth_method 0 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f12f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 1 -- 172.21.15.53:0/2148650200 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904802ac70 con 0x7f9058141cf0 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f905ce49640 10 monclient: dump: 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.663 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7000f040 con 0x7f3b78068f50 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b700157b0 con 0x7f3b78068f50 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b5effd640 10 monclient: dump: 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.664 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b7d64f640 5 monclient: authenticate success, global_id 4268 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348007320 0x7f33480dde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 1 -- 172.21.15.53:0/2636528188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348034570 msgr2=0x7f33480de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348034570 0x7f33480de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 1 monclient: found mon.noname-a 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 1 -- 172.21.15.53:0/2636528188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344009390 con 0x7f3348037df0 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f334f7fe640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3348034570 0x7f33480de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f334ffff640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3348007320 0x7f33480dde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f335493a640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3348037df0 0x7f334802d8e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3350066cc0 tx=0x7f3350064d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b7d64f640 10 monclient: _renew_subs 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b7d64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7814d6b0 con 0x7f3b78068f50 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f3b757fa640 4 set_mon_vals no callback set 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f3b7d64f640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f3b7d64f640 10 monclient: _renew_subs 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f3b7d64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7814d930 con 0x7f3b78068f50 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f3b7d64f640 1 librados: init done 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f333bfff640 10 client.?.objecter ms_handle_connect 0x7f3348037df0 2024-09-15T02:43:24.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f333bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01400cb70 con 0x7fa024069d30 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fa00b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 1 -- 172.21.15.53:0/1769696487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ff013d050 msgr2=0x7f0ff00a76a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 1 --2- 172.21.15.53:0/1769696487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ff013d050 0x7f0ff00a76a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 1 -- 172.21.15.53:0/1769696487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ff013cc80 msgr2=0x7f0ff0139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 1 --2- 172.21.15.53:0/1769696487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ff013cc80 0x7f0ff0139b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 1 monclient: found mon.noname-c 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7f0ff5285640 1 -- 172.21.15.53:0/1769696487 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8009a20 con 0x7f0ff013d930 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f905ee4d640 5 monclient: authenticate success, global_id 4303 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f905ee4d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f905ee4d640 4 set_mon_vals no callback set 2024-09-15T02:43:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 0x7fdacc0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc0adcd0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdacc0ae0a0 0x7fdacc13d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc0ae0a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc13da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc145df0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0a8dc0 con 0x7fdacc0adcd0 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0a8fc0 con 0x7fdacc0ae0a0 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0a92a0 con 0x7fdacc145df0 2024-09-15T02:43:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.534+0000 7fdad399d640 10 monclient(hunting): authenticate will time out at 741.104907s 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 1 -- 172.21.15.53:0/3127262760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f116406af90 msgr2=0x7f1164142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 1 --2- 172.21.15.53:0/3127262760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f116406af90 0x7f1164142f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 1 monclient: found mon.noname-b 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.533+0000 7f116359e640 1 -- 172.21.15.53:0/3127262760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1154009450 con 0x7f11641416a0 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f116359e640 1 --2- 172.21.15.53:0/3127262760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11641416a0 0x7f11641434c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f115800c650 tx=0x7f1158008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f114affd640 10 client.?.objecter ms_handle_connect 0x7f11641416a0 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f114affd640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa021ffb640 4 set_mon_vals no callback set 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014013470 con 0x7fa024069d30 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa00b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa00b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa00b7fe640 10 monclient: dump: 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa0097fa640 10 client.4300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa0097fa640 15 client.4300.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa0097fa640 20 client.4300.objecter _get_session osd=-1 returning homeless 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa0097fa640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa0097fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fa0097fa640 1 -- 172.21.15.53:0/849433396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa024063ff0 con 0x7fa024069d30 2024-09-15T02:43:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 1 --2- 172.21.15.53:0/2521790191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12fc145f00 0x7f12fc141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 10 monclient(hunting): get_auth_request con 0x7f12fc145f00 auth_method 0 2024-09-15T02:43:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f3b5cff9640 10 client.4268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f3b5cff9640 15 client.4268.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f3b5cff9640 20 client.4268.objecter _get_session osd=-1 returning homeless 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f3b5cff9640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f3b5cff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f3b5cff9640 1 -- 172.21.15.53:0/448214785 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b500028b0 con 0x7f3b78068f50 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350064320 con 0x7f3348037df0 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: dump: 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 5 monclient: authenticate success, global_id 4320 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 10 monclient: _renew_subs 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 1 -- 172.21.15.53:0/2636528188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c009390 con 0x7f3348037df0 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 10 monclient: _renew_subs 2024-09-15T02:43:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 1 -- 172.21.15.53:0/2636528188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334802dfc0 con 0x7f3348037df0 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f335639d640 1 librados: init done 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc13da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc13da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47116/0 (socket says 172.21.15.53:47116) 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 1 -- 172.21.15.53:0/183060002 learned_addr learned my addr 172.21.15.53:0/183060002 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 10 monclient(hunting): get_auth_request con 0x7fdacc145df0 auth_method 0 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33500644a0 con 0x7f3348037df0 2024-09-15T02:43:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335006e760 con 0x7f3348037df0 2024-09-15T02:43:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f333bfff640 10 monclient: dump: 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.681 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f334e7fc640 4 set_mon_vals no callback set 2024-09-15T02:43:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f114affd640 1 -- 172.21.15.53:0/3127262760 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158011600 con 0x7f11641416a0 2024-09-15T02:43:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 10 monclient: shutdown 2024-09-15T02:43:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 1 -- 172.21.15.53:0/2148650200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058141cf0 msgr2=0x7f9058142100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 1 --2- 172.21.15.53:0/2148650200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058141cf0 0x7f9058142100 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90580ad7d0 tx=0x7f90480090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 1 -- 172.21.15.53:0/2148650200 shutdown_connections 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 1 --2- 172.21.15.53:0/2148650200 >> v2:172.21.15.53:3300/0 conn(0x7f9058142640 0x7f90580aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 1 --2- 172.21.15.53:0/2148650200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058141cf0 0x7f9058142100 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 1 --2- 172.21.15.53:0/2148650200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 1 -- 172.21.15.53:0/2521790191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12fc145f00 msgr2=0x7f12fc141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 1 --2- 172.21.15.53:0/2521790191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12fc145f00 0x7f12fc141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 1 -- 172.21.15.53:0/2521790191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc06a8c0 msgr2=0x7f12fc06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 1 --2- 172.21.15.53:0/2521790191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12fc06a8c0 0x7f12fc06e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 1 monclient: found mon.noname-b 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7f12f9d9b640 1 -- 172.21.15.53:0/2521790191 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec009450 con 0x7f12fc145440 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12f9d9b640 1 --2- 172.21.15.53:0/2521790191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12fc145440 0x7f12fc159de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12f400ed60 tx=0x7f12f400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 10 client.?.objecter ms_handle_connect 0x7f12fc145440 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f114affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f114affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f905ee4d640 1 -- 172.21.15.53:0/2148650200 >> 172.21.15.53:0/2148650200 conn(0x7f9058067480 msgr2=0x7f9058067880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f114affd640 10 monclient: dump: 2024-09-15T02:43:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f114affd640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f114affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f114affd640 1 -- 172.21.15.53:0/3127262760 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1158011760 con 0x7f11641416a0 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 5 monclient: authenticate success, global_id 4309 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 10 monclient: _renew_subs 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11640a7210 con 0x7f11641416a0 2024-09-15T02:43:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f0ff5285640 1 --2- 172.21.15.53:0/1769696487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d930 0x7f0ff016e7f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009e60 tx=0x7f0fe8037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 10 monclient: _renew_subs 2024-09-15T02:43:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1164144c80 con 0x7f11641416a0 2024-09-15T02:43:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 client.?.objecter ms_handle_connect 0x7f0ff013d930 2024-09-15T02:43:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1739640 1 --2- 172.21.15.53:0/183060002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 0x7fdacc0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1739640 10 monclient(hunting): get_auth_request con 0x7fdacc0adcd0 auth_method 0 2024-09-15T02:43:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1739640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f1168908640 1 librados: init done 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.535+0000 7fdad1f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 1 -- 172.21.15.53:0/183060002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdacc0ae0a0 msgr2=0x7fdacc13d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 1 --2- 172.21.15.53:0/183060002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdacc0ae0a0 0x7fdacc13d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 1 -- 172.21.15.53:0/183060002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 msgr2=0x7fdacc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 1 --2- 172.21.15.53:0/183060002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 0x7fdacc0b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f400eec0 con 0x7f12fc145440 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 1 monclient: found mon.noname-c 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1f3a640 1 -- 172.21.15.53:0/183060002 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac8009390 con 0x7fdacc145df0 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7fdad1739640 1 --2- 172.21.15.53:0/183060002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdacc0adcd0 0x7fdacc0b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 10 monclient: dump: 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f400f040 con 0x7f12fc145440 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f12ebfff640 4 set_mon_vals no callback set 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.536+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f3b70015ba0 con 0x7f3b78068f50 2024-09-15T02:43:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.104:6816/32804 conn(0x7f3b6406e950 0x7f3b64070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f3b7008bc00 con 0x7f3b78068f50 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 10 client.4268.objecter ms_dispatch 0x7f3b78000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 3 client.4268.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 3 client.4268.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 10 client.4268.objecter checking command 1 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.53:6800/33157 conn(0x7f3b640748c0 0x7f3b64076cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 20 client.4268.objecter _get_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 20 client.4268.objecter put_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 15 client.4268.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 20 client.4268.objecter _get_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 20 client.4268.objecter get_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 15 client.4268.objecter _session_command_op_assign 1 1 2024-09-15T02:43:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 20 client.4268.objecter put_session s=0x7f3b64074210 osd=1 4 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 10 client.4268.objecter _send_command 1 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 --> v2:172.21.15.53:6800/33157 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b64077380 con 0x7f3b640748c0 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 20 client.4268.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b70009b10 con 0x7f3b78068f50 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f3b5effd640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f3b777fe640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.53:6800/33157 conn(0x7f3b640748c0 0x7f3b64076cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f3b777fe640 10 monclient: get_auth_request con 0x7f3b640748c0 auth_method 0 2024-09-15T02:43:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f905ee4d640 1 -- 172.21.15.53:0/2148650200 shutdown_connections 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f335639d640 10 client.4320.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f335639d640 15 client.4320.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f335639d640 20 client.4320.objecter _get_session osd=-1 returning homeless 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f335639d640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f335639d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f335639d640 1 -- 172.21.15.53:0/2636528188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33300028b0 con 0x7f3348037df0 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f4014480 con 0x7f12fc145440 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f12ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f12ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f12ea7fc640 10 monclient: dump: 2024-09-15T02:43:24.694 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7fdad1f3a640 1 --2- 172.21.15.53:0/183060002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc13da80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009db0 tx=0x7fdac400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7fdac17fa640 10 client.?.objecter ms_handle_connect 0x7fdacc145df0 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7fdac17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f114affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f114affd640 1 -- 172.21.15.53:0/3127262760 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158011b20 con 0x7f11641416a0 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f114affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f114affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f114affd640 10 monclient: dump: 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.696 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f116159a640 4 set_mon_vals no callback set 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac400cb30 con 0x7fdacc145df0 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:24.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: dump: 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fa014013820 con 0x7fa024069d30 2024-09-15T02:43:24.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f335008d020 con 0x7f3348037df0 2024-09-15T02:43:24.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fa00b7fe640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.104:6816/32804 conn(0x7fa01006e950 0x7fa010070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 5 monclient: authenticate success, global_id 4286 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 10 monclient: _renew_subs 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc0a9580 con 0x7fdacc145df0 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 10 monclient: _renew_subs 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc13dfc0 con 0x7fdacc145df0 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdad399d640 1 librados: init done 2024-09-15T02:43:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac400f040 con 0x7fdacc145df0 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4014440 con 0x7fdacc145df0 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac17fa640 10 monclient: dump: 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.702 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7fa01408bd50 con 0x7fa024069d30 2024-09-15T02:43:24.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fa022ffd640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.104:6816/32804 conn(0x7fa01006e950 0x7fa010070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f3b777fe640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.53:6800/33157 conn(0x7f3b640748c0 0x7f3b64076cf0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f3b76ffd640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.104:6816/32804 conn(0x7f3b6406e950 0x7f3b64070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f3b5effd640 10 client.4268.objecter ms_handle_connect 0x7f3b640748c0 2024-09-15T02:43:24.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f333bfff640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6816/32804 conn(0x7f333406e950 0x7f3334070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f114affd640 1 -- 172.21.15.53:0/3127262760 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f1158022350 con 0x7f11641416a0 2024-09-15T02:43:24.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== osd.1 v2:172.21.15.53:6800/33157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3b64077380 con 0x7f3b640748c0 2024-09-15T02:43:24.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f3b5effd640 10 client.4268.objecter ms_dispatch 0x7f3b78000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f3b5effd640 10 client.4268.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f3b5effd640 20 client.4268.objecter put_session s=0x7f3b64074210 osd=1 4 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f3b5effd640 15 client.4268.objecter _session_command_op_remove 1 1 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8009420 con 0x7f0ff013d930 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: dump: 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.704 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe8031a40 con 0x7f0ff013d930 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8031de0 con 0x7f0ff013d930 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0fecff9640 10 monclient: dump: 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 5 monclient: authenticate success, global_id 4280 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 10 monclient: _renew_subs 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff0067860 con 0x7f0ff013d930 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 10 monclient: _renew_subs 2024-09-15T02:43:24.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff00a6490 con 0x7f0ff013d930 2024-09-15T02:43:24.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.537+0000 7f0ff6ce8640 1 librados: init done 2024-09-15T02:43:24.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f0feeffd640 4 set_mon_vals no callback set 2024-09-15T02:43:24.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f905ee4d640 1 -- 172.21.15.53:0/2148650200 wait complete. 2024-09-15T02:43:24.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f335006d040 con 0x7f3348037df0 2024-09-15T02:43:24.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f333bfff640 10 client.4320.objecter ms_dispatch 0x7f3350000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f333bfff640 3 client.4320.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f333bfff640 3 client.4320.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7fdac37fe640 4 set_mon_vals no callback set 2024-09-15T02:43:24.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f333bfff640 10 client.4320.objecter checking command 1 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f334ffff640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6816/32804 conn(0x7f333406e950 0x7f3334070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6812/33348 conn(0x7f33340748c0 0x7f3334076cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f334ffff640 10 monclient: get_auth_request con 0x7f333406e950 auth_method 0 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 20 client.4320.objecter _get_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 20 client.4320.objecter put_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 15 client.4320.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 20 client.4320.objecter _get_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 20 client.4320.objecter get_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 15 client.4320.objecter _session_command_op_assign 4 1 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 20 client.4320.objecter put_session s=0x7f3334074210 osd=4 4 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 10 client.4320.objecter _send_command 1 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 --> v2:172.21.15.104:6812/33348 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3334077380 con 0x7f33340748c0 2024-09-15T02:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 20 client.4320.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33500ccaa0 con 0x7f3348037df0 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f333bfff640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f1300eef640 5 monclient: authenticate success, global_id 4315 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f1300eef640 10 monclient: _renew_subs 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f1300eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.538+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12fc15a380 con 0x7f12fc145440 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f1300eef640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f1300eef640 10 monclient: _renew_subs 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f1300eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12fc141f90 con 0x7f12fc145440 2024-09-15T02:43:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f1300eef640 1 librados: init done 2024-09-15T02:43:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fdad399d640 10 client.4286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fdad399d640 15 client.4286.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fdad399d640 20 client.4286.objecter _get_session osd=-1 returning homeless 2024-09-15T02:43:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fdad399d640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fdad399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdaac0028b0 con 0x7fdacc145df0 2024-09-15T02:43:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f114affd640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6816/32804 conn(0x7f115006e950 0x7f1150070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa022ffd640 10 monclient: get_auth_request con 0x7fa01006e950 auth_method 0 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 10 client.4300.objecter ms_dispatch 0x7fa024000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 3 client.4300.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 3 client.4300.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 10 client.4300.objecter checking command 1 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa022ffd640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.104:6816/32804 conn(0x7fa01006e950 0x7fa010070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa02406cfc0 tx=0x7fa018007e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.53:6802/33158 conn(0x7fa0100748c0 0x7fa010076cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 20 client.4300.objecter _get_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 20 client.4300.objecter put_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 15 client.4300.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 20 client.4300.objecter _get_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 20 client.4300.objecter get_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 15 client.4300.objecter _session_command_op_assign 2 1 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 20 client.4300.objecter put_session s=0x7fa010074210 osd=2 4 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 10 client.4300.objecter _send_command 1 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 --> v2:172.21.15.53:6802/33158 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa010077380 con 0x7fa0100748c0 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 20 client.4300.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 10 client.4300.objecter ms_handle_connect 0x7fa01006e950 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa01408c080 con 0x7fa024069d30 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7fa00b7fe640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fa023fff640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.53:6802/33158 conn(0x7fa0100748c0 0x7fa010076cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fa023fff640 10 monclient: get_auth_request con 0x7fa0100748c0 auth_method 0 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fa00b7fe640 10 client.4300.objecter ms_handle_connect 0x7fa0100748c0 2024-09-15T02:43:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7fa023fff640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.53:6802/33158 conn(0x7fa0100748c0 0x7fa010076cf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f905ee4d640 10 monclient: build_initial_monmap 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.539+0000 7f905ee4d640 10 monclient: monmap: 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f905ee4d640 1 librados: starting msgr at 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f905ee4d640 1 librados: starting objecter 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f905ee4d640 1 Processor -- start 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f1162d9d640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6816/32804 conn(0x7f115006e950 0x7f1150070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f114affd640 1 -- 172.21.15.53:0/3127262760 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f115808b720 con 0x7f11641416a0 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f114affd640 10 client.4309.objecter ms_dispatch 0x7f1164000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f114affd640 3 client.4309.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f114affd640 3 client.4309.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f1162d9d640 10 monclient: get_auth_request con 0x7f115006e950 auth_method 0 2024-09-15T02:43:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.540+0000 7f334ffff640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6816/32804 conn(0x7f333406e950 0x7f3334070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f333c034000 tx=0x7f333c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f333bfff640 10 client.4320.objecter ms_handle_connect 0x7f333406e950 2024-09-15T02:43:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f114affd640 20 client.4309.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f1148ff9640 10 client.4309.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f0ff6ce8640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f0ff6ce8640 15 client.4280.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f0ff6ce8640 20 client.4280.objecter _get_session osd=-1 returning homeless 2024-09-15T02:43:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f0ff6ce8640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f334f7fe640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6812/33348 conn(0x7f33340748c0 0x7f3334076cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f3b76ffd640 10 monclient: get_auth_request con 0x7f3b6406e950 auth_method 0 2024-09-15T02:43:24.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f905ee4d640 1 -- start start 2024-09-15T02:43:24.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f905ee4d640 1 librados: setting wanted keys 2024-09-15T02:43:24.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f905ee4d640 1 librados: calling monclient init 2024-09-15T02:43:24.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f905ee4d640 10 monclient: init 2024-09-15T02:43:24.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== osd.2 v2:172.21.15.53:6802/33158 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa010077380 con 0x7fa0100748c0 2024-09-15T02:43:24.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f0ff6ce8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:24.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0fd00028b0 con 0x7f0ff013d930 2024-09-15T02:43:24.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7fa00b7fe640 10 client.4300.objecter ms_dispatch 0x7fa024000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f0fe8003040 con 0x7f0ff013d930 2024-09-15T02:43:24.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7fa00b7fe640 10 client.4300.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7fa00b7fe640 20 client.4300.objecter put_session s=0x7fa010074210 osd=2 4 2024-09-15T02:43:24.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.541+0000 7fa00b7fe640 15 client.4300.objecter _session_command_op_remove 2 1 2024-09-15T02:43:24.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f334f7fe640 10 monclient: get_auth_request con 0x7f33340748c0 auth_method 0 2024-09-15T02:43:24.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fdac4014830 con 0x7fdacc145df0 2024-09-15T02:43:24.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:24.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:24.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f1148ff9640 15 client.4309.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): picked mon.noname-c con 0x7f90580ad160 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9058141cf0 0x7f9058151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): picked mon.noname-a con 0x7f9058141cf0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058142640 0x7f9058151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): picked mon.noname-b con 0x7f9058142640 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9058152200 con 0x7f9058141cf0 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9058142f00 con 0x7f9058142640 2024-09-15T02:43:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9058144900 con 0x7f90580ad160 2024-09-15T02:43:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f905ee4d640 10 monclient(hunting): authenticate will time out at 741.112592s 2024-09-15T02:43:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f1148ff9640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6804/33347 conn(0x7f116413e4d0 0x7f11641552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f1162d9d640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6816/32804 conn(0x7f115006e950 0x7f1150070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1154009210 tx=0x7f1154032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f114affd640 10 client.4309.objecter ms_handle_connect 0x7f115006e950 2024-09-15T02:43:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f0fecff9640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.104:6816/32804 conn(0x7f0fd406e950 0x7f0fd4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f0fe80ad460 con 0x7f0ff013d930 2024-09-15T02:43:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f0fecff9640 10 client.4280.objecter ms_dispatch 0x7f0ff0000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f1148ff9640 20 client.4309.objecter _get_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f0fecff9640 3 client.4280.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f0fecff9640 3 client.4280.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f1148ff9640 20 client.4309.objecter put_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f1148ff9640 20 client.4309.objecter _get_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f1148ff9640 15 client.4309.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1148ff9640 20 client.4309.objecter get_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1148ff9640 15 client.4309.objecter _session_command_op_assign 7 1 2024-09-15T02:43:24.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6816/32804 conn(0x7fdab006e880 0x7fdab0070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058142640 0x7f9058151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1148ff9640 20 client.4309.objecter put_session s=0x7f1164063ff0 osd=7 4 2024-09-15T02:43:24.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1148ff9640 10 client.4309.objecter _send_command 1 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1300eef640 10 client.4315.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1300eef640 15 client.4315.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1300eef640 20 client.4315.objecter _get_session osd=-1 returning homeless 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1300eef640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1300eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f12fc0646e0 con 0x7f12fc145440 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f1148ff9640 1 -- 172.21.15.53:0/3127262760 --> v2:172.21.15.104:6804/33347 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11640a6020 con 0x7f116413e4d0 2024-09-15T02:43:24.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058142640 0x7f9058151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60262/0 (socket says 172.21.15.53:60262) 2024-09-15T02:43:24.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 -- 172.21.15.53:0/2312728387 learned_addr learned my addr 172.21.15.53:0/2312728387 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:24.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): get_auth_request con 0x7f9058142640 auth_method 0 2024-09-15T02:43:24.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f3b76ffd640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.104:6816/32804 conn(0x7f3b6406e950 0x7f3b64070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800b8d0 tx=0x7f3b68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 1 --2- 172.21.15.53:0/2312728387 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 10 monclient(hunting): get_auth_request con 0x7f90580ad160 auth_method 0 2024-09-15T02:43:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.542+0000 7f3b5effd640 10 client.4268.objecter ms_handle_connect 0x7f3b6406e950 2024-09-15T02:43:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0ff4a84640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.104:6816/32804 conn(0x7f0fd406e950 0x7f0fd4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdad1739640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6816/32804 conn(0x7fdab006e880 0x7fdab0070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdad1739640 10 monclient: get_auth_request con 0x7fdab006e880 auth_method 0 2024-09-15T02:43:24.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f12f4014870 con 0x7f12fc145440 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0ff4a84640 10 monclient: get_auth_request con 0x7f0fd406e950 auth_method 0 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905d64a640 1 --2- 172.21.15.53:0/2312728387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9058141cf0 0x7f9058151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905de4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905d64a640 10 monclient(hunting): get_auth_request con 0x7f9058141cf0 auth_method 0 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905d64a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:24.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905d64a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905d64a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7fdac408cc30 con 0x7fdacc145df0 2024-09-15T02:43:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 10 client.4286.objecter ms_dispatch 0x7fdacc000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 3 client.4286.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 3 client.4286.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 10 client.4286.objecter checking command 1 2024-09-15T02:43:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 10 client.4280.objecter checking command 1 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.53:6801/33156 conn(0x7f0fd4074750 0x7f0fd4076b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 20 client.4280.objecter _get_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 20 client.4280.objecter put_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 15 client.4280.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 20 client.4280.objecter _get_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 20 client.4280.objecter get_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 15 client.4280.objecter _session_command_op_assign 0 1 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 20 client.4280.objecter put_session s=0x7f0fe8009420 osd=0 4 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 10 client.4280.objecter _send_command 1 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 --> v2:172.21.15.53:6801/33156 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fd4077210 con 0x7f0fd4074750 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 20 client.4280.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fe80af940 con 0x7f0ff013d930 2024-09-15T02:43:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f0fecff9640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 -- 172.21.15.53:0/2312728387 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 msgr2=0x7f905806e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 --2- 172.21.15.53:0/2312728387 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90580ad160 0x7f905806e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 -- 172.21.15.53:0/2312728387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9058141cf0 msgr2=0x7f9058151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 --2- 172.21.15.53:0/2312728387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9058141cf0 0x7f9058151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient: _finish_hunting 0 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 monclient: found mon.noname-b 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f905e64c640 1 -- 172.21.15.53:0/2312728387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9050009a20 con 0x7f9058142640 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6808/33357 conn(0x7fdab00747f0 0x7fdab0076c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 20 client.4286.objecter _get_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 20 client.4286.objecter put_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 15 client.4286.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 20 client.4286.objecter _get_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 20 client.4286.objecter get_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 15 client.4286.objecter _session_command_op_assign 6 1 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 20 client.4286.objecter put_session s=0x7fdab0074140 osd=6 4 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 10 client.4286.objecter _send_command 1 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 --> v2:172.21.15.104:6808/33357 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdab00772b0 con 0x7fdab00747f0 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdac17fa640 20 client.4286.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7fdad1739640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6816/32804 conn(0x7fdab006e880 0x7fdab0070d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdac8034000 tx=0x7fdac8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6816/32804 conn(0x7f12dc06e850 0x7f12dc070d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdac408cf60 con 0x7fdacc145df0 2024-09-15T02:43:24.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7fdac17fa640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-15T02:43:24.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7fdac17fa640 10 client.4286.objecter ms_handle_connect 0x7fdab006e880 2024-09-15T02:43:24.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7fdad0f38640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6808/33357 conn(0x7fdab00747f0 0x7fdab0076c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7fdad0f38640 10 monclient: get_auth_request con 0x7fdab00747f0 auth_method 0 2024-09-15T02:43:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f0ff4a84640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.104:6816/32804 conn(0x7f0fd406e950 0x7f0fd4070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc005840 tx=0x7f0fdc007940 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f0fecff9640 10 client.4280.objecter ms_handle_connect 0x7f0fd406e950 2024-09-15T02:43:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f0feffff640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.53:6801/33156 conn(0x7f0fd4074750 0x7f0fd4076b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905d64a640 1 --2- 172.21.15.53:0/2312728387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9058141cf0 0x7f9058151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:43:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905e64c640 1 --2- 172.21.15.53:0/2312728387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058142640 0x7f9058151cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9050009f20 tx=0x7f905000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f116259c640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6804/33347 conn(0x7f116413e4d0 0x7f11641552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.543+0000 7f116259c640 10 monclient: get_auth_request con 0x7f116413e4d0 auth_method 0 2024-09-15T02:43:24.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 client.?.objecter ms_handle_connect 0x7f9058142640 2024-09-15T02:43:24.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12fa59c640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6816/32804 conn(0x7f12dc06e850 0x7f12dc070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:24.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12fa59c640 10 monclient: get_auth_request con 0x7f12dc06e850 auth_method 0 2024-09-15T02:43:24.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f12f408c280 con 0x7f12fc145440 2024-09-15T02:43:24.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 10 client.4315.objecter ms_dispatch 0x7f12fc000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 3 client.4315.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 3 client.4315.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 1 -- 172.21.15.53:0/2312728387 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905000ce10 con 0x7f9058142640 2024-09-15T02:43:24.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 10 client.4315.objecter checking command 1 2024-09-15T02:43:24.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6800/33346 conn(0x7f12dc074650 0x7f12dc076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f116259c640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6804/33347 conn(0x7f116413e4d0 0x7f11641552f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: dump: 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:24.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 5 monclient: authenticate success, global_id 4318 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 10 monclient: _renew_subs 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 20 client.4315.objecter _get_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 20 client.4315.objecter put_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 15 client.4315.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 20 client.4315.objecter _get_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 20 client.4315.objecter get_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 15 client.4315.objecter _session_command_op_assign 5 1 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 20 client.4315.objecter put_session s=0x7f12f400f040 osd=5 4 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 10 client.4315.objecter _send_command 1 2024-09-15T02:43:24.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 --> v2:172.21.15.104:6800/33346 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12dc077110 con 0x7f12dc074650 2024-09-15T02:43:24.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 20 client.4315.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12f408e890 con 0x7f12fc145440 2024-09-15T02:43:24.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7fdac17fa640 10 client.4286.objecter ms_handle_connect 0x7fdab00747f0 2024-09-15T02:43:24.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7fdad0f38640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6808/33357 conn(0x7fdab00747f0 0x7fdab0076c20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f0feffff640 10 monclient: get_auth_request con 0x7f0fd4074750 auth_method 0 2024-09-15T02:43:24.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 1 -- 172.21.15.53:0/2312728387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9048009390 con 0x7f9058142640 2024-09-15T02:43:24.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 1 -- 172.21.15.53:0/2312728387 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905000f040 con 0x7f9058142640 2024-09-15T02:43:24.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:24.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12ea7fc640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-15T02:43:24.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 1 -- 172.21.15.53:0/2312728387 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90500146b0 con 0x7f9058142640 2024-09-15T02:43:24.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12fad9d640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6800/33346 conn(0x7f12dc074650 0x7f12dc076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f12fad9d640 10 monclient: get_auth_request con 0x7f12dc074650 auth_method 0 2024-09-15T02:43:24.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f334f7fe640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6812/33348 conn(0x7f33340748c0 0x7f3334076cf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f333bfff640 10 client.4320.objecter ms_handle_connect 0x7f33340748c0 2024-09-15T02:43:24.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f12ea7fc640 10 client.4315.objecter ms_handle_connect 0x7f12dc06e850 2024-09-15T02:43:24.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f12fa59c640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6816/32804 conn(0x7f12dc06e850 0x7f12dc070d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12ec032680 tx=0x7f12ec032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904dffb640 10 monclient: dump: 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:24.746 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f904f7fe640 4 set_mon_vals no callback set 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 10 monclient: _renew_subs 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 1 -- 172.21.15.53:0/2312728387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90581436e0 con 0x7f9058142640 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.544+0000 7f905ee4d640 1 librados: init done 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f12fad9d640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6800/33346 conn(0x7f12dc074650 0x7f12dc076a80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f12ea7fc640 10 client.4315.objecter ms_handle_connect 0x7f12dc074650 2024-09-15T02:43:24.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f114affd640 10 client.4309.objecter ms_handle_connect 0x7f116413e4d0 2024-09-15T02:43:24.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f114affd640 1 -- 172.21.15.53:0/3127262760 <== osd.7 v2:172.21.15.104:6804/33347 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f11640a6020 con 0x7f116413e4d0 2024-09-15T02:43:24.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== osd.6 v2:172.21.15.104:6808/33357 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdab00772b0 con 0x7fdab00747f0 2024-09-15T02:43:24.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7fdac17fa640 10 client.4286.objecter ms_dispatch 0x7fdacc000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7fdac17fa640 10 client.4286.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7fdac17fa640 20 client.4286.objecter put_session s=0x7fdab0074140 osd=6 4 2024-09-15T02:43:24.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7fdac17fa640 15 client.4286.objecter _session_command_op_remove 6 1 2024-09-15T02:43:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f114affd640 10 client.4309.objecter ms_dispatch 0x7f1164000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f114affd640 10 client.4309.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f114affd640 20 client.4309.objecter put_session s=0x7f1164063ff0 osd=7 4 2024-09-15T02:43:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== osd.4 v2:172.21.15.104:6812/33348 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3334077380 con 0x7f33340748c0 2024-09-15T02:43:24.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f333bfff640 10 client.4320.objecter ms_dispatch 0x7f3350000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f114affd640 15 client.4309.objecter _session_command_op_remove 7 1 2024-09-15T02:43:24.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f333bfff640 10 client.4320.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f333bfff640 20 client.4320.objecter put_session s=0x7f3334074210 osd=4 4 2024-09-15T02:43:24.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f333bfff640 15 client.4320.objecter _session_command_op_remove 4 1 2024-09-15T02:43:24.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f0feffff640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.53:6801/33156 conn(0x7f0fd4074750 0x7f0fd4076b80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.545+0000 7f0fecff9640 10 client.4280.objecter ms_handle_connect 0x7f0fd4074750 2024-09-15T02:43:24.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.546+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== osd.5 v2:172.21.15.104:6800/33346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f12dc077110 con 0x7f12dc074650 2024-09-15T02:43:24.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.546+0000 7f12ea7fc640 10 client.4315.objecter ms_dispatch 0x7f12fc000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.546+0000 7f12ea7fc640 10 client.4315.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.546+0000 7f12ea7fc640 20 client.4315.objecter put_session s=0x7f12f400f040 osd=5 4 2024-09-15T02:43:24.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.546+0000 7f12ea7fc640 15 client.4315.objecter _session_command_op_remove 5 1 2024-09-15T02:43:24.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.547+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== osd.0 v2:172.21.15.53:6801/33156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0fd4077210 con 0x7f0fd4074750 2024-09-15T02:43:24.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.547+0000 7f0fecff9640 10 client.4280.objecter ms_dispatch 0x7f0ff0000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.547+0000 7f0fecff9640 10 client.4280.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.547+0000 7f0fecff9640 20 client.4280.objecter put_session s=0x7f0fe8009420 osd=0 4 2024-09-15T02:43:24.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.547+0000 7f0fecff9640 15 client.4280.objecter _session_command_op_remove 0 1 2024-09-15T02:43:24.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.547+0000 7f904dffb640 1 -- 172.21.15.53:0/2312728387 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f9050003020 con 0x7f9058142640 2024-09-15T02:43:24.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.548+0000 7f904dffb640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.104:6816/32804 conn(0x7f903c06e950 0x7f903c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.548+0000 7f904dffb640 1 -- 172.21.15.53:0/2312728387 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f905008c160 con 0x7f9058142640 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.548+0000 7f904dffb640 10 client.4318.objecter ms_dispatch 0x7f9058000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.548+0000 7f904dffb640 3 client.4318.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.548+0000 7f904dffb640 3 client.4318.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.548+0000 7f904dffb640 20 client.4318.objecter dump_active .. 0 homeless 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 15 client.4318.objecter _session_command_op_assign -1 1 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.53:6812/33159 conn(0x7f9058154840 0x7f9058154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 20 client.4318.objecter _get_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 20 client.4318.objecter put_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 20 client.4318.objecter _get_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 15 client.4318.objecter _session_command_op_remove -1 1 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 20 client.4318.objecter get_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 15 client.4318.objecter _session_command_op_assign 3 1 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 20 client.4318.objecter put_session s=0x7f90580646e0 osd=3 4 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 10 client.4318.objecter _send_command 1 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.549+0000 7f905ee4d640 1 -- 172.21.15.53:0/2312728387 --> v2:172.21.15.53:6812/33159 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f905806c140 con 0x7f9058154840 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.550+0000 7f905d64a640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.53:6812/33159 conn(0x7f9058154840 0x7f9058154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.550+0000 7f905d64a640 10 monclient: get_auth_request con 0x7f9058154840 auth_method 0 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.550+0000 7f904dffb640 10 client.4318.objecter ms_handle_connect 0x7f9058154840 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.550+0000 7f905d64a640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.53:6812/33159 conn(0x7f9058154840 0x7f9058154c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.551+0000 7f905de4b640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.104:6816/32804 conn(0x7f903c06e950 0x7f903c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.551+0000 7f904dffb640 1 -- 172.21.15.53:0/2312728387 <== osd.3 v2:172.21.15.53:6812/33159 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f905806c140 con 0x7f9058154840 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.551+0000 7f904dffb640 10 client.4318.objecter ms_dispatch 0x7f9058000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.551+0000 7f904dffb640 10 client.4318.objecter _finish_command 1 = system:0 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.551+0000 7f904dffb640 20 client.4318.objecter put_session s=0x7f90580646e0 osd=3 4 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.551+0000 7f904dffb640 15 client.4318.objecter _session_command_op_remove 3 1 2024-09-15T02:43:24.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.553+0000 7f905de4b640 10 monclient: get_auth_request con 0x7f903c06e950 auth_method 0 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 10 client.4300.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 15 client.4300.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 20 client.4300.objecter _get_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 20 client.4300.objecter put_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 20 client.4300.objecter _get_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 15 client.4300.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 20 client.4300.objecter get_session s=0x7fa010074210 osd=2 3 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 15 client.4300.objecter _session_command_op_assign 2 2 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 20 client.4300.objecter put_session s=0x7fa010074210 osd=2 4 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 10 client.4300.objecter _send_command 2 2024-09-15T02:43:24.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.570+0000 7fa0097fa640 1 -- 172.21.15.53:0/849433396 --> v2:172.21.15.53:6802/33158 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa024063ff0 con 0x7fa0100748c0 2024-09-15T02:43:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.568+0000 7f904dffb640 10 client.4318.objecter ms_handle_connect 0x7f903c06e950 2024-09-15T02:43:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.568+0000 7f905de4b640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.104:6816/32804 conn(0x7f903c06e950 0x7f903c070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f905400bc10 tx=0x7f9054008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa00b7fe640 1 -- 172.21.15.53:0/849433396 <== osd.2 v2:172.21.15.53:6802/33158 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa024063ff0 con 0x7fa0100748c0 2024-09-15T02:43:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa00b7fe640 10 client.4300.objecter ms_dispatch 0x7fa024000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa00b7fe640 10 client.4300.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa00b7fe640 20 client.4300.objecter put_session s=0x7fa010074210 osd=2 4 2024-09-15T02:43:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa00b7fe640 15 client.4300.objecter _session_command_op_remove 2 2 2024-09-15T02:43:24.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 10 librados: watch_flush enter 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 10 librados: watch_flush exit 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 10 client.4300.objecter close_session for osd.2 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 >> v2:172.21.15.53:6802/33158 conn(0x7fa0100748c0 msgr2=0x7fa010076cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.53:6802/33158 conn(0x7fa0100748c0 0x7fa010076cf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 20 client.4300.objecter put_session s=0x7fa010074210 osd=2 1 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 20 client.4300.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.571+0000 7fa029edd640 10 client.4300.objecter successfully canceled tick 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 >> v2:172.21.15.104:6816/32804 conn(0x7fa01006e950 msgr2=0x7fa010070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 --2- 172.21.15.53:0/849433396 >> v2:172.21.15.104:6816/32804 conn(0x7fa01006e950 0x7fa010070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa02406cfc0 tx=0x7fa018007e20 comp rx=0 tx=0).stop 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 10 monclient: shutdown 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa024069d30 msgr2=0x7fa02406e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 --2- 172.21.15.53:0/849433396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa024069d30 0x7fa02406e030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa01400ea30 tx=0x7fa0140091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa023fff640 1 -- 172.21.15.53:0/849433396 reap_dead start 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 shutdown_connections 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 >> 172.21.15.53:0/849433396 conn(0x7fa0240b4400 msgr2=0x7fa02406aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 shutdown_connections 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 10 client.4268.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 15 client.4268.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 20 client.4268.objecter _get_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 20 client.4268.objecter put_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 20 client.4268.objecter _get_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 15 client.4268.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 20 client.4268.objecter get_session s=0x7f3b64074210 osd=1 3 2024-09-15T02:43:24.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 15 client.4268.objecter _session_command_op_assign 1 2 2024-09-15T02:43:24.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 20 client.4268.objecter put_session s=0x7f3b64074210 osd=1 4 2024-09-15T02:43:24.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 10 client.4268.objecter _send_command 2 2024-09-15T02:43:24.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7f3b5cff9640 1 -- 172.21.15.53:0/448214785 --> v2:172.21.15.53:6800/33157 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3b500011c0 con 0x7f3b640748c0 2024-09-15T02:43:24.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.572+0000 7fa029edd640 1 -- 172.21.15.53:0/849433396 wait complete. 2024-09-15T02:43:24.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fa029edd640 1 librados: shutdown 2024-09-15T02:43:24.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 10 client.4320.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 15 client.4320.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 20 client.4320.objecter _get_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 20 client.4320.objecter put_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 20 client.4320.objecter _get_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 15 client.4320.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 20 client.4320.objecter get_session s=0x7f3334074210 osd=4 3 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 15 client.4320.objecter _session_command_op_assign 4 2 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 20 client.4320.objecter put_session s=0x7f3334074210 osd=4 4 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 10 client.4320.objecter _send_command 2 2024-09-15T02:43:24.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f335639d640 1 -- 172.21.15.53:0/2636528188 --> v2:172.21.15.104:6812/33348 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33300011c0 con 0x7f33340748c0 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f3b5effd640 1 -- 172.21.15.53:0/448214785 <== osd.1 v2:172.21.15.53:6800/33157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3b500011c0 con 0x7f3b640748c0 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f3b5effd640 10 client.4268.objecter ms_dispatch 0x7f3b78000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f3b5effd640 10 client.4268.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f3b5effd640 20 client.4268.objecter put_session s=0x7f3b64074210 osd=1 4 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 10 client.4286.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 15 client.4286.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 20 client.4286.objecter _get_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 20 client.4286.objecter put_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 20 client.4286.objecter _get_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 15 client.4286.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 20 client.4286.objecter get_session s=0x7fdab0074140 osd=6 3 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 15 client.4286.objecter _session_command_op_assign 6 2 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 20 client.4286.objecter put_session s=0x7fdab0074140 osd=6 4 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 10 client.4286.objecter _send_command 2 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 --> v2:172.21.15.104:6808/33357 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdaac0011e0 con 0x7fdab00747f0 2024-09-15T02:43:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.573+0000 7f3b5effd640 15 client.4268.objecter _session_command_op_remove 1 2 2024-09-15T02:43:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f333bfff640 1 -- 172.21.15.53:0/2636528188 <== osd.4 v2:172.21.15.104:6812/33348 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f33300011c0 con 0x7f33340748c0 2024-09-15T02:43:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f333bfff640 10 client.4320.objecter ms_dispatch 0x7f3350000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f333bfff640 10 client.4320.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 10 librados: watch_flush enter 2024-09-15T02:43:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f333bfff640 20 client.4320.objecter put_session s=0x7f3334074210 osd=4 4 2024-09-15T02:43:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f333bfff640 15 client.4320.objecter _session_command_op_remove 4 2 2024-09-15T02:43:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 10 client.4315.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 15 client.4315.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 20 client.4315.objecter _get_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 20 client.4315.objecter put_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 20 client.4315.objecter _get_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 15 client.4315.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 20 client.4315.objecter get_session s=0x7f12f400f040 osd=5 3 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 15 client.4315.objecter _session_command_op_assign 5 2 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 20 client.4315.objecter put_session s=0x7f12f400f040 osd=5 4 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 10 client.4315.objecter _send_command 2 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 --> v2:172.21.15.104:6800/33346 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f12fc0ac540 con 0x7f12dc074650 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdac17fa640 1 -- 172.21.15.53:0/183060002 <== osd.6 v2:172.21.15.104:6808/33357 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdaac0011e0 con 0x7fdab00747f0 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdac17fa640 10 client.4286.objecter ms_dispatch 0x7fdacc000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdac17fa640 10 client.4286.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdac17fa640 20 client.4286.objecter put_session s=0x7fdab0074140 osd=6 4 2024-09-15T02:43:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdac17fa640 15 client.4286.objecter _session_command_op_remove 6 2 2024-09-15T02:43:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 10 librados: watch_flush exit 2024-09-15T02:43:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 10 client.4268.objecter close_session for osd.1 2024-09-15T02:43:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 >> v2:172.21.15.53:6800/33157 conn(0x7f3b640748c0 msgr2=0x7f3b64076cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.53:6800/33157 conn(0x7f3b640748c0 0x7f3b64076cf0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3339ffb640 10 librados: watch_flush enter 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 10 librados: watch_flush enter 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 10 librados: watch_flush exit 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 10 client.4286.objecter close_session for osd.6 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6808/33357 conn(0x7fdab00747f0 msgr2=0x7fdab0076c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6808/33357 conn(0x7fdab00747f0 0x7fdab0076c20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 20 client.4286.objecter put_session s=0x7fdab0074140 osd=6 1 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 20 client.4286.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 10 client.4286.objecter successfully canceled tick 2024-09-15T02:43:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 20 client.4268.objecter put_session s=0x7f3b64074210 osd=1 1 2024-09-15T02:43:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6816/32804 conn(0x7fdab006e880 msgr2=0x7fdab0070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 1 --2- 172.21.15.53:0/183060002 >> v2:172.21.15.104:6816/32804 conn(0x7fdab006e880 0x7fdab0070d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdac8034000 tx=0x7fdac8032000 comp rx=0 tx=0).stop 2024-09-15T02:43:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 10 monclient: shutdown 2024-09-15T02:43:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 msgr2=0x7fdacc13da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad399d640 1 --2- 172.21.15.53:0/183060002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdacc145df0 0x7fdacc13da80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009db0 tx=0x7fdac400b040 comp rx=0 tx=0).stop 2024-09-15T02:43:24.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7fdad1f3a640 1 -- 172.21.15.53:0/183060002 reap_dead start 2024-09-15T02:43:24.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 20 client.4268.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 10 librados: watch_flush exit 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 10 client.4320.objecter close_session for osd.4 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 1 -- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6812/33348 conn(0x7f33340748c0 msgr2=0x7f3334076cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6812/33348 conn(0x7f33340748c0 0x7f3334076cf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 10 client.4268.objecter successfully canceled tick 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 >> v2:172.21.15.104:6816/32804 conn(0x7f3b6406e950 msgr2=0x7f3b64070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.574+0000 7f3b7d64f640 1 --2- 172.21.15.53:0/448214785 >> v2:172.21.15.104:6816/32804 conn(0x7f3b6406e950 0x7f3b64070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800b8d0 tx=0x7f3b68008040 comp rx=0 tx=0).stop 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b7d64f640 10 monclient: shutdown 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b7d64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b78068f50 msgr2=0x7f3b781428e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b7d64f640 1 --2- 172.21.15.53:0/448214785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b78068f50 0x7f3b781428e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000d180 tx=0x7f3b700099d0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b777fe640 1 -- 172.21.15.53:0/448214785 reap_dead start 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 shutdown_connections 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 >> 172.21.15.53:0/448214785 conn(0x7f3b78062d90 msgr2=0x7f3b7813f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 20 client.4320.objecter put_session s=0x7f3334074210 osd=4 1 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 20 client.4320.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 10 client.4320.objecter successfully canceled tick 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f12ea7fc640 1 -- 172.21.15.53:0/2521790191 <== osd.5 v2:172.21.15.104:6800/33346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f12fc0ac540 con 0x7f12dc074650 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f12ea7fc640 10 client.4315.objecter ms_dispatch 0x7f12fc000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f12ea7fc640 10 client.4315.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f12ea7fc640 20 client.4315.objecter put_session s=0x7f12f400f040 osd=5 4 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f12ea7fc640 15 client.4315.objecter _session_command_op_remove 5 2 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 10 client.4309.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 15 client.4309.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 20 client.4309.objecter _get_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 20 client.4309.objecter put_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 20 client.4309.objecter _get_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 15 client.4309.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 20 client.4309.objecter get_session s=0x7f1164063ff0 osd=7 3 2024-09-15T02:43:24.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 15 client.4309.objecter _session_command_op_assign 7 2 2024-09-15T02:43:24.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 20 client.4309.objecter put_session s=0x7f1164063ff0 osd=7 4 2024-09-15T02:43:24.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 10 client.4309.objecter _send_command 2 2024-09-15T02:43:24.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1148ff9640 1 -- 172.21.15.53:0/3127262760 --> v2:172.21.15.104:6804/33347 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1164142070 con 0x7f116413e4d0 2024-09-15T02:43:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 1 -- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6816/32804 conn(0x7f333406e950 msgr2=0x7f3334070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 1 --2- 172.21.15.53:0/2636528188 >> v2:172.21.15.104:6816/32804 conn(0x7f333406e950 0x7f3334070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f333c034000 tx=0x7f333c032000 comp rx=0 tx=0).stop 2024-09-15T02:43:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 10 monclient: shutdown 2024-09-15T02:43:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 1 -- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3348037df0 msgr2=0x7f334802d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3339ffb640 1 --2- 172.21.15.53:0/2636528188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3348037df0 0x7f334802d8e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3350066cc0 tx=0x7f3350064d50 comp rx=0 tx=0).stop 2024-09-15T02:43:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 10 librados: watch_flush enter 2024-09-15T02:43:24.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 shutdown_connections 2024-09-15T02:43:24.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 10 librados: watch_flush exit 2024-09-15T02:43:24.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 10 client.4315.objecter close_session for osd.5 2024-09-15T02:43:24.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6800/33346 conn(0x7f12dc074650 msgr2=0x7f12dc076a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6800/33346 conn(0x7f12dc074650 0x7f12dc076a80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f114affd640 1 -- 172.21.15.53:0/3127262760 <== osd.7 v2:172.21.15.104:6804/33347 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1164142070 con 0x7f116413e4d0 2024-09-15T02:43:24.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f114affd640 10 client.4309.objecter ms_dispatch 0x7f1164000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 20 client.4315.objecter put_session s=0x7f12f400f040 osd=5 1 2024-09-15T02:43:24.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f114affd640 10 client.4309.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f114affd640 20 client.4309.objecter put_session s=0x7f1164063ff0 osd=7 4 2024-09-15T02:43:24.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f114affd640 15 client.4309.objecter _session_command_op_remove 7 2 2024-09-15T02:43:24.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 20 client.4315.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f1300eef640 10 client.4315.objecter successfully canceled tick 2024-09-15T02:43:24.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 >> 172.21.15.53:0/183060002 conn(0x7fdacc062d90 msgr2=0x7fdacc0b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 shutdown_connections 2024-09-15T02:43:24.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f3b7d64f640 1 -- 172.21.15.53:0/448214785 wait complete. 2024-09-15T02:43:24.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 shutdown_connections 2024-09-15T02:43:24.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f335493a640 1 -- 172.21.15.53:0/2636528188 reap_dead start 2024-09-15T02:43:24.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f3b7d64f640 1 librados: shutdown 2024-09-15T02:43:24.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f3339ffb640 1 -- 172.21.15.53:0/2636528188 shutdown_connections 2024-09-15T02:43:24.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f3339ffb640 1 -- 172.21.15.53:0/2636528188 >> 172.21.15.53:0/2636528188 conn(0x7f3348039ef0 msgr2=0x7f3348006940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 10 librados: watch_flush enter 2024-09-15T02:43:24.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 10 librados: watch_flush exit 2024-09-15T02:43:24.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 10 client.4309.objecter close_session for osd.7 2024-09-15T02:43:24.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6804/33347 conn(0x7f116413e4d0 msgr2=0x7f11641552f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6804/33347 conn(0x7f116413e4d0 0x7f11641552f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 20 client.4309.objecter put_session s=0x7f1164063ff0 osd=7 1 2024-09-15T02:43:24.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 20 client.4309.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 10 client.4309.objecter successfully canceled tick 2024-09-15T02:43:24.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7fdad399d640 1 -- 172.21.15.53:0/183060002 wait complete. 2024-09-15T02:43:24.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.575+0000 7fdad399d640 1 librados: shutdown 2024-09-15T02:43:24.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6816/32804 conn(0x7f115006e950 msgr2=0x7f1150070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f3339ffb640 1 -- 172.21.15.53:0/2636528188 shutdown_connections 2024-09-15T02:43:24.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f3339ffb640 1 -- 172.21.15.53:0/2636528188 wait complete. 2024-09-15T02:43:24.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6816/32804 conn(0x7f12dc06e850 msgr2=0x7f12dc070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1300eef640 1 --2- 172.21.15.53:0/2521790191 >> v2:172.21.15.104:6816/32804 conn(0x7f12dc06e850 0x7f12dc070d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12ec032680 tx=0x7f12ec032000 comp rx=0 tx=0).stop 2024-09-15T02:43:24.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 1 --2- 172.21.15.53:0/3127262760 >> v2:172.21.15.104:6816/32804 conn(0x7f115006e950 0x7f1150070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1154009210 tx=0x7f1154032000 comp rx=0 tx=0).stop 2024-09-15T02:43:24.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 10 monclient: shutdown 2024-09-15T02:43:24.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11641416a0 msgr2=0x7f11641434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1168908640 1 --2- 172.21.15.53:0/3127262760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11641416a0 0x7f11641434c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f115800c650 tx=0x7f1158008040 comp rx=0 tx=0).stop 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 15 client.4280.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 20 client.4280.objecter _get_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 20 client.4280.objecter put_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 20 client.4280.objecter _get_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 15 client.4280.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 20 client.4280.objecter get_session s=0x7f0fe8009420 osd=0 3 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 15 client.4280.objecter _session_command_op_assign 0 2 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 20 client.4280.objecter put_session s=0x7f0fe8009420 osd=0 4 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 10 client.4280.objecter _send_command 2 2024-09-15T02:43:24.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 --> v2:172.21.15.53:6801/33156 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0fd00011c0 con 0x7f0fd4074750 2024-09-15T02:43:24.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1300eef640 10 monclient: shutdown 2024-09-15T02:43:24.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1300eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12fc145440 msgr2=0x7f12fc159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f116359e640 1 -- 172.21.15.53:0/3127262760 reap_dead start 2024-09-15T02:43:24.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f1300eef640 1 --2- 172.21.15.53:0/2521790191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12fc145440 0x7f12fc159de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12f400ed60 tx=0x7f12f400b040 comp rx=0 tx=0).stop 2024-09-15T02:43:24.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.576+0000 7f12fad9d640 1 -- 172.21.15.53:0/2521790191 reap_dead start 2024-09-15T02:43:24.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 shutdown_connections 2024-09-15T02:43:24.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 >> 172.21.15.53:0/2521790191 conn(0x7f12fc067b60 msgr2=0x7f12fc0ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f3339ffb640 1 librados: shutdown 2024-09-15T02:43:24.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 shutdown_connections 2024-09-15T02:43:24.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 shutdown_connections 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1300eef640 1 -- 172.21.15.53:0/2521790191 wait complete. 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1300eef640 1 librados: shutdown 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 >> 172.21.15.53:0/3127262760 conn(0x7f11640b0510 msgr2=0x7f11640aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f0fecff9640 1 -- 172.21.15.53:0/1769696487 <== osd.0 v2:172.21.15.53:6801/33156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0fd00011c0 con 0x7f0fd4074750 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f0fecff9640 10 client.4280.objecter ms_dispatch 0x7f0ff0000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f0fecff9640 10 client.4280.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f0fecff9640 20 client.4280.objecter put_session s=0x7f0fe8009420 osd=0 4 2024-09-15T02:43:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f0fecff9640 15 client.4280.objecter _session_command_op_remove 0 2 2024-09-15T02:43:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f0ff6ce8640 10 librados: watch_flush enter 2024-09-15T02:43:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 shutdown_connections 2024-09-15T02:43:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1168908640 1 -- 172.21.15.53:0/3127262760 wait complete. 2024-09-15T02:43:24.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.577+0000 7f1168908640 1 librados: shutdown 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 10 librados: watch_flush exit 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 10 client.4280.objecter close_session for osd.0 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 >> v2:172.21.15.53:6801/33156 conn(0x7f0fd4074750 msgr2=0x7f0fd4076b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.53:6801/33156 conn(0x7f0fd4074750 0x7f0fd4076b80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 20 client.4280.objecter put_session s=0x7f0fe8009420 osd=0 1 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 20 client.4280.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 10 client.4280.objecter successfully canceled tick 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 >> v2:172.21.15.104:6816/32804 conn(0x7f0fd406e950 msgr2=0x7f0fd4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 --2- 172.21.15.53:0/1769696487 >> v2:172.21.15.104:6816/32804 conn(0x7f0fd406e950 0x7f0fd4070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc005840 tx=0x7f0fdc007940 comp rx=0 tx=0).stop 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 10 monclient: shutdown 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d930 msgr2=0x7f0ff016e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 --2- 172.21.15.53:0/1769696487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ff013d930 0x7f0ff016e7f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009e60 tx=0x7f0fe8037000 comp rx=0 tx=0).stop 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff5285640 1 -- 172.21.15.53:0/1769696487 reap_dead start 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 shutdown_connections 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.578+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 >> 172.21.15.53:0/1769696487 conn(0x7f0ff0067280 msgr2=0x7f0ff00aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 shutdown_connections 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f0ff6ce8640 1 -- 172.21.15.53:0/1769696487 wait complete. 2024-09-15T02:43:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f0ff6ce8640 1 librados: shutdown 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 15 client.4318.objecter _session_command_op_assign -1 2 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 20 client.4318.objecter _get_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 20 client.4318.objecter put_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 20 client.4318.objecter _get_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 15 client.4318.objecter _session_command_op_remove -1 2 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 20 client.4318.objecter get_session s=0x7f90580646e0 osd=3 3 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 15 client.4318.objecter _session_command_op_assign 3 2 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 20 client.4318.objecter put_session s=0x7f90580646e0 osd=3 4 2024-09-15T02:43:24.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 10 client.4318.objecter _send_command 2 2024-09-15T02:43:24.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.579+0000 7f905ee4d640 1 -- 172.21.15.53:0/2312728387 --> v2:172.21.15.53:6812/33159 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9058155390 con 0x7f9058154840 2024-09-15T02:43:24.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.580+0000 7f904dffb640 1 -- 172.21.15.53:0/2312728387 <== osd.3 v2:172.21.15.53:6812/33159 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9058155390 con 0x7f9058154840 2024-09-15T02:43:24.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.580+0000 7f904dffb640 10 client.4318.objecter ms_dispatch 0x7f9058000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:43:24.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.580+0000 7f904dffb640 10 client.4318.objecter _finish_command 2 = system:0 2024-09-15T02:43:24.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.580+0000 7f904dffb640 20 client.4318.objecter put_session s=0x7f90580646e0 osd=3 4 2024-09-15T02:43:24.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.580+0000 7f904dffb640 15 client.4318.objecter _session_command_op_remove 3 2 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.580+0000 7f90437fe640 10 librados: watch_flush enter 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 10 librados: watch_flush exit 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 10 client.4318.objecter close_session for osd.3 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 1 -- 172.21.15.53:0/2312728387 >> v2:172.21.15.53:6812/33159 conn(0x7f9058154840 msgr2=0x7f9058154c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.53:6812/33159 conn(0x7f9058154840 0x7f9058154c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 20 client.4318.objecter put_session s=0x7f90580646e0 osd=3 1 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 20 client.4318.objecter shutdown clearing up homeless session... 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 10 client.4318.objecter successfully canceled tick 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 1 -- 172.21.15.53:0/2312728387 >> v2:172.21.15.104:6816/32804 conn(0x7f903c06e950 msgr2=0x7f903c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 1 --2- 172.21.15.53:0/2312728387 >> v2:172.21.15.104:6816/32804 conn(0x7f903c06e950 0x7f903c070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f905400bc10 tx=0x7f9054008040 comp rx=0 tx=0).stop 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 10 monclient: shutdown 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 1 -- 172.21.15.53:0/2312728387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058142640 msgr2=0x7f9058151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f90437fe640 1 --2- 172.21.15.53:0/2312728387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9058142640 0x7f9058151cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9050009f20 tx=0x7f905000b040 comp rx=0 tx=0).stop 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.581+0000 7f905e64c640 1 -- 172.21.15.53:0/2312728387 reap_dead start 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.582+0000 7f90437fe640 1 -- 172.21.15.53:0/2312728387 shutdown_connections 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.582+0000 7f90437fe640 1 -- 172.21.15.53:0/2312728387 >> 172.21.15.53:0/2312728387 conn(0x7f9058067480 msgr2=0x7f905806a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.582+0000 7f90437fe640 1 -- 172.21.15.53:0/2312728387 shutdown_connections 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.582+0000 7f90437fe640 1 -- 172.21.15.53:0/2312728387 wait complete. 2024-09-15T02:43:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:24.586+0000 7f90437fe640 1 librados: shutdown 2024-09-15T02:43:24.801 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:24.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T02:43:24.802 INFO:teuthology.orchestra.run.smithi053.stdout:55834574851 2024-09-15T02:43:24.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T02:43:24.803 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:24.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T02:43:24.804 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:24.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T02:43:24.805 INFO:teuthology.orchestra.run.smithi053.stdout:55834574851 2024-09-15T02:43:24.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T02:43:24.806 INFO:teuthology.orchestra.run.smithi053.stdout:55834574853 2024-09-15T02:43:24.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T02:43:24.807 INFO:teuthology.orchestra.run.smithi053.stdout:55834574851 2024-09-15T02:43:24.807 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T02:43:24.808 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:24.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.005+0000 7fafdcc57640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.005+0000 7fafdcc57640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.005+0000 7fafdcc57640 10 monclient: monmap: 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.006520+0000 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.006520+0000 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.009 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7fafdcc57640 1 Processor -- start 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7fafdcc57640 1 -- start start 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7fafdcc57640 10 monclient: init 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafd81398b0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafd813a560 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fafd806a8c0 0x7fafd806b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient(hunting): picked mon.noname-c con 0x7fafd806a8c0 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd806ade0 con 0x7fafd813a560 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd806cd20 con 0x7fafd81398b0 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd806bec0 con 0x7fafd806a8c0 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafdcc57640 10 monclient(hunting): authenticate will time out at 741.579494s 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7ff6fe9dd640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7ff6fe9dd640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7ff6fe9dd640 10 monclient: monmap: 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.009518+0000 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.009518+0000 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.012 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafd5d9b640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fafd813a560 0x7fafd806b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40736/0 (socket says 172.21.15.53:40736) 2024-09-15T02:43:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fafd5d9b640 1 -- 172.21.15.53:0/1402669208 learned_addr learned my addr 172.21.15.53:0/1402669208 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 1 --2- 172.21.15.53:0/1402669208 >> v2:172.21.15.104:3300/0 conn(0x7fafd806a8c0 0x7fafd806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): get_auth_request con 0x7fafd806a8c0 auth_method 0 2024-09-15T02:43:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7fa4b8109640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7f6a6399d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7f6a6399d640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7f6a6399d640 10 monclient: monmap: 2024-09-15T02:43:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.010746+0000 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.010746+0000 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.015 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7f7e40a4f640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7f7e40a4f640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7f7e40a4f640 10 monclient: monmap: 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.011204+0000 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.011204+0000 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.016 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 1 -- 172.21.15.53:0/1402669208 >> v1:172.21.15.104:6789/0 conn(0x7fafd81398b0 legacy=0x7fafd806ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 1 -- 172.21.15.53:0/1402669208 >> v1:172.21.15.53:6789/0 conn(0x7fafd813a560 legacy=0x7fafd806b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7fafd6d9d640 1 -- 172.21.15.53:0/1402669208 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd81374a0 con 0x7fafd806a8c0 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.008+0000 7fa4b8109640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.009+0000 7fa4b8109640 10 monclient: monmap: 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.009779+0000 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.009779+0000 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.019 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.020 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.010+0000 7ff6fe9dd640 1 Processor -- start 2024-09-15T02:43:25.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7f7e40a4f640 1 Processor -- start 2024-09-15T02:43:25.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7f7e40a4f640 1 -- start start 2024-09-15T02:43:25.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7f7e40a4f640 10 monclient: init 2024-09-15T02:43:25.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7f6a6399d640 1 Processor -- start 2024-09-15T02:43:25.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7ff6fe9dd640 1 -- start start 2024-09-15T02:43:25.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7ff6fe9dd640 10 monclient: init 2024-09-15T02:43:25.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7fa4b8109640 1 Processor -- start 2024-09-15T02:43:25.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7f6a6399d640 1 -- start start 2024-09-15T02:43:25.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.011+0000 7f6a6399d640 10 monclient: init 2024-09-15T02:43:25.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T02:43:25.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7fa4b8109640 1 -- start start 2024-09-15T02:43:25.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7fa4b8109640 10 monclient: init 2024-09-15T02:43:25.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6f00c63b0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff6f00c6d20 0x7ff6f00304d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f00c6d20 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:25.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6f0007090 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:25.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e40a4f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00340e0 con 0x7ff6f0007090 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0032850 con 0x7ff6f00c63b0 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00c6780 con 0x7ff6f00c6d20 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe9dd640 10 monclient(hunting): authenticate will time out at 741.582193s 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e40a4f640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T02:43:25.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e40a4f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7e3c06d700 0x7f7e3c0acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fd1da640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff6f00c6d20 0x7ff6f00304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fd1da640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff6f00c6d20 0x7ff6f00304d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51752/0 (socket says 172.21.15.53:51752) 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe1dc640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff6f0007090 0x7ff6f0030a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40742/0 (socket says 172.21.15.53:40742) 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fe1dc640 1 -- 172.21.15.53:0/665032328 learned_addr learned my addr 172.21.15.53:0/665032328 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fd1da640 10 monclient(hunting): get_auth_request con 0x7ff6f00c6d20 auth_method 0 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fd1da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e40a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3c06d700 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e40a4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e3c06d090 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e40a4f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06dad0 0x7f7e3c138b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e3a59c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7e3c06d700 0x7f7e3c0acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e39d9b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7e3c06d090 0x7f7e3c13c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.53:50650/0 (socket says 172.21.15.53:50650) 2024-09-15T02:43:25.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e3a59c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7e3c06d700 0x7f7e3c0acc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51744/0 (socket says 172.21.15.53:51744) 2024-09-15T02:43:25.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fd1da640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7ff6fd1da640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f6a6399d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f6a6399d640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f6a6399d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a5c0b0870 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f6a6399d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a5c0b0c40 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a5c0abcb0 0x7f6a5c1491d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a5c0abcb0 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a60f38640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6a5c0b0c40 0x7f6a5c0a9d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40746/0 (socket says 172.21.15.53:40746) 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a60f38640 1 -- 172.21.15.53:0/1270537309 learned_addr learned my addr 172.21.15.53:0/1270537309 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 1 --2- 172.21.15.53:0/1270537309 >> v2:172.21.15.104:3300/0 conn(0x7f6a5c0abcb0 0x7f6a5c1491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070203572 0 0) 0x7ff6f0032850 con 0x7ff6f00c63b0 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): my global_id is 4327 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e40034e0 con 0x7ff6f00c63b0 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4b00a98d0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4b00a9ca0 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa4b0141300 0x7fa4b01416f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b0141300 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b006b4d0 con 0x7fa4b00a9ca0 2024-09-15T02:43:25.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0140e70 con 0x7fa4b00a98d0 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00ac990 con 0x7fa4b0141300 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b8109640 10 monclient(hunting): authenticate will time out at 741.583189s 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e3ad9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06dad0 0x7f7e3c138b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e3ad9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06dad0 0x7f7e3c138b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47134/0 (socket says 172.21.15.53:47134) 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7f7e39d9b640 1 -- 172.21.15.53:0/110494396 learned_addr learned my addr 172.21.15.53:0/110494396 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:25.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3c06dad0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c06efd0 con 0x7f7e3c06d090 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3c13c7d0 con 0x7f7e3c06d700 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c06f7d0 con 0x7f7e3c06dad0 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e40a4f640 10 monclient(hunting): authenticate will time out at 741.583319s 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7fd8b386f640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7fd8b386f640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.012+0000 7fd8b386f640 10 monclient: monmap: 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.013114+0000 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.013114+0000 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.031 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5c0a98d0 con 0x7f6a5c0b0c40 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 1 -- 172.21.15.53:0/1270537309 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5c06b420 con 0x7f6a5c0b0870 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 1 -- 172.21.15.53:0/1270537309 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a5c13b5a0 con 0x7f6a5c0abcb0 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a6399d640 10 monclient(hunting): authenticate will time out at 741.583338s 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): get_auth_request con 0x7f6a5c0abcb0 auth_method 0 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7a931ed640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7a931ed640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7a931ed640 10 monclient: monmap: 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.014119+0000 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.014119+0000 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.033 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.034 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fd1da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fd1da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fd1da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b5ea5640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa4b00a98d0 0x7fa4b00ac280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.53:50666/0 (socket says 172.21.15.53:50666) 2024-09-15T02:43:25.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b5ea5640 1 -- 172.21.15.53:0/4227499138 learned_addr learned my addr 172.21.15.53:0/4227499138 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:43:25.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3a59c640 10 monclient(hunting): get_auth_request con 0x7f7e3c06d700 auth_method 0 2024-09-15T02:43:25.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 1 --2- 172.21.15.53:0/4227499138 >> v2:172.21.15.104:3300/0 conn(0x7fa4b0141300 0x7fa4b01416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2408582480 0 0) 0x7ff6e40034e0 con 0x7ff6f00c63b0 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 >> v2:172.21.15.53:3300/0 conn(0x7ff6f00c6d20 msgr2=0x7ff6f00304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 --2- 172.21.15.53:0/665032328 >> v2:172.21.15.53:3300/0 conn(0x7ff6f00c6d20 0x7ff6f00304d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 >> v1:172.21.15.53:6789/0 conn(0x7ff6f0007090 legacy=0x7ff6f0030a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.104:6789/0 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f002c770 con 0x7ff6f00c63b0 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3ad9d640 10 monclient(hunting): get_auth_request con 0x7f7e3c06dad0 auth_method 0 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): get_auth_request con 0x7fa4b0141300 auth_method 0 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f7e3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2024-09-15T02:43:25.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 65778851 0 0) 0x7ff6e8001450 con 0x7ff6f00c63b0 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: dump: 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.041 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 >> v1:172.21.15.104:6789/0 conn(0x7ff6f00c63b0 legacy=0x7ff6f00339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 1 -- 172.21.15.53:0/1270537309 >> v1:172.21.15.104:6789/0 conn(0x7f6a5c0b0870 legacy=0x7f6a5c141410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T02:43:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6e4003a10 0x7ff6e4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-09-15T02:43:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 1 -- 172.21.15.53:0/4227499138 >> v1:172.21.15.104:6789/0 conn(0x7fa4b00a98d0 legacy=0x7fa4b00ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 1 -- 172.21.15.53:0/4227499138 >> v1:172.21.15.53:6789/0 conn(0x7fa4b00a9ca0 legacy=0x7fa4b013e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 1 -- 172.21.15.53:0/1270537309 >> v1:172.21.15.53:6789/0 conn(0x7f6a5c0b0c40 legacy=0x7f6a5c0a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): picked mon.c con 0x7ff6e4003a10 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e4004780 0x7ff6e4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): picked mon.a con 0x7ff6e4004780 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e4005130 0x7ff6e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): picked mon.b con 0x7ff6e4005130 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00c6780 con 0x7ff6e4004780 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e400c0a0 con 0x7ff6e4005130 2024-09-15T02:43:25.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e400c260 con 0x7ff6e4003a10 2024-09-15T02:43:25.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7ff6fc9d9640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:43:25.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e4004780 0x7ff6e4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): get_auth_request con 0x7ff6e4004780 auth_method 0 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e4005130 0x7ff6e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd9db640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6e4003a10 0x7ff6e4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 10 monclient(hunting): get_auth_request con 0x7ff6e4005130 auth_method 0 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd9db640 10 monclient(hunting): get_auth_request con 0x7ff6e4003a10 auth_method 0 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd9db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd9db640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd9db640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:25.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:25.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7fa4b66a6640 1 -- 172.21.15.53:0/4227499138 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b013b580 con 0x7fa4b0141300 2024-09-15T02:43:25.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7fd8b386f640 1 Processor -- start 2024-09-15T02:43:25.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7a931ed640 1 Processor -- start 2024-09-15T02:43:25.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7fafd6d9d640 1 --2- 172.21.15.53:0/1402669208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd806a8c0 0x7fafd806b980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafd000af70 tx=0x7fafd000a890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed6cdb23559e4870 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 1 -- 172.21.15.53:0/110494396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06dad0 msgr2=0x7f7e3c138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 1 --2- 172.21.15.53:0/110494396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06dad0 0x7f7e3c138b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 1 -- 172.21.15.53:0/110494396 >> v1:172.21.15.104:6789/0 conn(0x7f7e3c06d090 legacy=0x7f7e3c13c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 1 monclient: found mon.noname-a 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7e3a59c640 1 -- 172.21.15.53:0/110494396 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c06f3f0 con 0x7f7e3c06d700 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient(hunting): handle_auth_done global_id 4327 payload 293 2024-09-15T02:43:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 1 -- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6e4003a10 msgr2=0x7ff6e4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6e4003a10 0x7ff6e4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 1 -- 172.21.15.53:0/665032328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e4005130 msgr2=0x7ff6e400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e4005130 0x7ff6e400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fe1dc640 1 -- 172.21.15.53:0/665032328 reap_dead start 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 1 monclient: found mon.a 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 1 -- 172.21.15.53:0/665032328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f002c770 con 0x7ff6e4004780 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7ff6fd1da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.013+0000 7f6a61f3a640 1 -- 172.21.15.53:0/1270537309 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c06b700 con 0x7f6a5c0abcb0 2024-09-15T02:43:25.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a61f3a640 1 --2- 172.21.15.53:0/1270537309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0abcb0 0x7f6a5c1491d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009e90 tx=0x7f6a54009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93fa47e85927260a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 1 -- 172.21.15.53:0/1402669208 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd000a2e0 con 0x7fafd806a8c0 2024-09-15T02:43:25.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: dump: 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fa4b66a6640 1 --2- 172.21.15.53:0/4227499138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 0x7fa4b01416f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0097d0 tx=0x7fa4ac0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd4e6e1bdb06f777 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 1 -- 172.21.15.53:0/1402669208 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd0008450 con 0x7fafd806a8c0 2024-09-15T02:43:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafdcc57640 5 monclient: authenticate success, global_id 4321 2024-09-15T02:43:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafdcc57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 1 -- 172.21.15.53:0/1270537309 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54007550 con 0x7f6a5c0abcb0 2024-09-15T02:43:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 1 -- 172.21.15.53:0/1402669208 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0008830 con 0x7fafd806a8c0 2024-09-15T02:43:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: dump: 2024-09-15T02:43:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 1 -- 172.21.15.53:0/1270537309 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5402d050 con 0x7f6a5c0abcb0 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 1 -- 172.21.15.53:0/1270537309 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a540078e0 con 0x7f6a5c0abcb0 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafdcc57640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafdcc57640 4 set_mon_vals no callback set 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a53fff640 10 monclient: dump: 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a6399d640 5 monclient: authenticate success, global_id 4333 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a6399d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f6a6399d640 4 set_mon_vals no callback set 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fafd559a640 10 monclient: dump: 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.060 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.061 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7ff6fe9dd640 5 monclient: authenticate success, global_id 4327 2024-09-15T02:43:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7ff6fe9dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fd8b386f640 1 -- start start 2024-09-15T02:43:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7fd8b386f640 10 monclient: init 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f7e3a59c640 1 --2- 172.21.15.53:0/110494396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06d700 0x7f7e3c0acc80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009850 tx=0x7f7e2c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118966e4615eff31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f67e9f5d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f67e9f5d640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f67e9f5d640 10 monclient: monmap: 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.016658+0000 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.016658+0000 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.063 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7a931ed640 1 -- start start 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 1 -- 172.21.15.53:0/4227499138 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac02aaf0 con 0x7fa4b0141300 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 10 monclient: dump: 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.065 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.014+0000 7f7a931ed640 10 monclient: init 2024-09-15T02:43:25.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b8109640 5 monclient: authenticate success, global_id 4339 2024-09-15T02:43:25.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b8109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 1 -- 172.21.15.53:0/4227499138 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac02ac50 con 0x7fa4b0141300 2024-09-15T02:43:25.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b4ea3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f6a6399d640 10 monclient: shutdown 2024-09-15T02:43:25.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fafdcc57640 10 monclient: shutdown 2024-09-15T02:43:25.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fafdcc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fafdcc57640 1 -- 172.21.15.53:0/1402669208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd806a8c0 msgr2=0x7fafd806b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fafdcc57640 1 --2- 172.21.15.53:0/1402669208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd806a8c0 0x7fafd806b980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafd000af70 tx=0x7fafd000a890 comp rx=0 tx=0).stop 2024-09-15T02:43:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f7a931ed640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.015+0000 7f7a931ed640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T02:43:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7a840073b0 0x7f7a84032c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): picked mon.noname-a con 0x7f7a840073b0 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a84008740 0x7f7a84035be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): picked mon.noname-b con 0x7f7a84008740 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a84008b70 0x7f7a84036120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): picked mon.noname-c con 0x7f7a84008b70 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a840331b0 con 0x7f7a840073b0 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8402ec10 con 0x7f7a84008b70 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84009780 con 0x7f7a84008740 2024-09-15T02:43:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a931ed640 10 monclient(hunting): authenticate will time out at 741.586141s 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a84008b70 0x7f7a84036120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a84008b70 0x7f7a84036120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60306/0 (socket says 172.21.15.53:60306) 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 1 -- 172.21.15.53:0/1852455683 learned_addr learned my addr 172.21.15.53:0/1852455683 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 10 monclient(hunting): get_auth_request con 0x7f7a84008b70 auth_method 0 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f6a6399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 1 -- 172.21.15.53:0/110494396 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c036440 con 0x7f7e3c06d700 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: dump: 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 1 -- 172.21.15.53:0/110494396 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c0365a0 con 0x7f7e3c06d700 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 1 -- 172.21.15.53:0/110494396 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c036940 con 0x7f7e3c06d700 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e3959a640 10 monclient: dump: 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e40a4f640 5 monclient: authenticate success, global_id 4347 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e40a4f640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7e40a4f640 4 set_mon_vals no callback set 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f6a6399d640 1 -- 172.21.15.53:0/1270537309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0abcb0 msgr2=0x7f6a5c1491d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f6a6399d640 1 --2- 172.21.15.53:0/1270537309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0abcb0 0x7f6a5c1491d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009e90 tx=0x7f6a54009780 comp rx=0 tx=0).stop 2024-09-15T02:43:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fd8b386f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fd8b386f640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T02:43:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fd8b386f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd8ac0b0aa0 0x7fd8ac142200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fd8b386f640 10 monclient(hunting): picked mon.noname-c con 0x7fd8ac0b0aa0 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fd8b386f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 0x7fd8ac145750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fd8b386f640 10 monclient(hunting): picked mon.noname-b con 0x7fd8ac145340 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fd8b386f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8ac145c90 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f6a6399d640 1 -- 172.21.15.53:0/1270537309 shutdown_connections 2024-09-15T02:43:25.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd8ac0b0aa0 0x7fd8ac142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b386f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ac142a30 con 0x7fd8ac145c90 2024-09-15T02:43:25.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd8ac0b0aa0 0x7fd8ac142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60318/0 (socket says 172.21.15.53:60318) 2024-09-15T02:43:25.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7f7a929ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2024-09-15T02:43:25.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f6a6399d640 1 --2- 172.21.15.53:0/1270537309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0abcb0 0x7f6a5c1491d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f6a6399d640 1 -- 172.21.15.53:0/1270537309 >> 172.21.15.53:0/1270537309 conn(0x7f6a5c067b00 msgr2=0x7f6a5c0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 1 -- 172.21.15.53:0/1930949188 learned_addr learned my addr 172.21.15.53:0/1930949188 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b1e0c640 1 --1- 172.21.15.53:0/1930949188 >> v1:172.21.15.104:6789/0 conn(0x7fd8ac145c90 0x7fd8ac0aab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.53:50682/0 (socket says 172.21.15.53:50682) 2024-09-15T02:43:25.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 1 -- 172.21.15.53:0/1852455683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a84008740 msgr2=0x7f7a84035be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 1 --2- 172.21.15.53:0/1852455683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a84008740 0x7f7a84035be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 1 -- 172.21.15.53:0/1852455683 >> v2:172.21.15.53:3300/0 conn(0x7f7a840073b0 msgr2=0x7f7a84032c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a921eb640 1 --2- 172.21.15.53:0/1852455683 >> v2:172.21.15.53:3300/0 conn(0x7f7a840073b0 0x7f7a84032c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b386f640 1 -- 172.21.15.53:0/1930949188 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ac142740 con 0x7fd8ac0b0aa0 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b386f640 1 -- 172.21.15.53:0/1930949188 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0b0e70 con 0x7fd8ac145340 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b386f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b386f640 10 monclient(hunting): authenticate will time out at 741.587164s 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fafdcc57640 1 -- 172.21.15.53:0/1402669208 shutdown_connections 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fafdcc57640 1 --2- 172.21.15.53:0/1402669208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd806a8c0 0x7fafd806b980 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fafdcc57640 1 -- 172.21.15.53:0/1402669208 >> 172.21.15.53:0/1402669208 conn(0x7fafd8067b60 msgr2=0x7fafd80b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 10 monclient(hunting): get_auth_request con 0x7fd8ac0b0aa0 auth_method 0 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b8109640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.016+0000 7fa4b8109640 4 set_mon_vals no callback set 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b4ea3640 1 -- 172.21.15.53:0/4227499138 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac032440 con 0x7fa4b0141300 2024-09-15T02:43:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b4ea3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b4ea3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b4ea3640 10 monclient: dump: 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.079 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 1 --2- 172.21.15.53:0/1852455683 >> v2:172.21.15.53:3300/0 conn(0x7f7a840073b0 0x7f7a84032c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:43:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a929ec640 1 -- 172.21.15.53:0/1852455683 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840082b0 con 0x7f7a84008b70 2024-09-15T02:43:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 10 monclient: shutdown 2024-09-15T02:43:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 1 -- 172.21.15.53:0/4227499138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 msgr2=0x7fa4b01416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 1 --2- 172.21.15.53:0/4227499138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 0x7fa4b01416f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0097d0 tx=0x7fa4ac0090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 0x7fd8ac145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a921eb640 10 monclient: get_auth_request con 0x7f7a840073b0 auth_method 0 2024-09-15T02:43:25.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7a921eb640 1 --2- 172.21.15.53:0/1852455683 >> v2:172.21.15.53:3300/0 conn(0x7f7a840073b0 0x7f7a84032c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:43:25.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b0e0a640 10 monclient(hunting): get_auth_request con 0x7fd8ac145340 auth_method 0 2024-09-15T02:43:25.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b0e0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fd1da640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e4004780 0x7ff6e4004bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4005500 tx=0x7ff6f40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a392f53d7c006d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 10 monclient: shutdown 2024-09-15T02:43:25.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fafdcc57640 1 -- 172.21.15.53:0/1402669208 shutdown_connections 2024-09-15T02:43:25.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06d700 msgr2=0x7f7e3c0acc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384530063 0 0) 0x7fd8ac142a30 con 0x7fd8ac145c90 2024-09-15T02:43:25.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/110494396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06d700 0x7f7e3c0acc80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009850 tx=0x7f7e2c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b0e0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b0e0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 1 -- 172.21.15.53:0/4227499138 shutdown_connections 2024-09-15T02:43:25.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 1 --2- 172.21.15.53:0/4227499138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 0x7fa4b01416f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 1 -- 172.21.15.53:0/4227499138 >> 172.21.15.53:0/4227499138 conn(0x7fa4b0067b60 msgr2=0x7fa4b00b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8a3fff640 10 monclient(hunting): my global_id is 4348 2024-09-15T02:43:25.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 1 -- 172.21.15.53:0/4227499138 shutdown_connections 2024-09-15T02:43:25.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 shutdown_connections 2024-09-15T02:43:25.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/110494396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06dad0 0x7f7e3c138b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/110494396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06d700 0x7f7e3c0acc80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 >> 172.21.15.53:0/110494396 conn(0x7f7e3c063810 msgr2=0x7f7e3c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8980034e0 con 0x7fd8ac145c90 2024-09-15T02:43:25.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7f67e9f5d640 1 Processor -- start 2024-09-15T02:43:25.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f402aaa0 con 0x7ff6e4004780 2024-09-15T02:43:25.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:25.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 10 monclient: dump: 2024-09-15T02:43:25.087 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f402ac00 con 0x7ff6e4004780 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 1 -- 172.21.15.53:0/665032328 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4032440 con 0x7ff6e4004780 2024-09-15T02:43:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fc9d9640 10 monclient: dump: 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fe9dd640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7ff6fe9dd640 4 set_mon_vals no callback set 2024-09-15T02:43:25.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 shutdown_connections 2024-09-15T02:43:25.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952951503 0 0) 0x7fd8980034e0 con 0x7fd8ac145c90 2024-09-15T02:43:25.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7f7e40a4f640 1 -- 172.21.15.53:0/110494396 wait complete. 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 msgr2=0x7fd8ac145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 0x7fd8ac145750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 >> v2:172.21.15.104:3300/0 conn(0x7fd8ac0b0aa0 msgr2=0x7fd8ac142200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 --2- 172.21.15.53:0/1930949188 >> v2:172.21.15.104:3300/0 conn(0x7fd8ac0b0aa0 0x7fd8ac142200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:43:25.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.104:6789/0 2024-09-15T02:43:25.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac0aca00 con 0x7fd8ac145c90 2024-09-15T02:43:25.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7f6a6399d640 1 -- 172.21.15.53:0/1270537309 shutdown_connections 2024-09-15T02:43:25.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 65778851 0 0) 0x7fd8a40017f0 con 0x7fd8ac145c90 2024-09-15T02:43:25.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7f6a6399d640 1 -- 172.21.15.53:0/1270537309 wait complete. 2024-09-15T02:43:25.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8b386f640 5 monclient: authenticate success, global_id 4348 2024-09-15T02:43:25.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8b386f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8b160b640 1 --2- 172.21.15.53:0/1930949188 >> v2:172.21.15.104:3300/0 conn(0x7fd8ac0b0aa0 0x7fd8ac142200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: dump: 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.096 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-15T02:43:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 >> v1:172.21.15.104:6789/0 conn(0x7fd8ac145c90 legacy=0x7fd8ac0aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7f7a929ec640 1 --2- 172.21.15.53:0/1852455683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008b70 0x7f7a84036120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a84032320 tx=0x7f7a880091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d30fa7c58563f893 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd898003a50 0x7fd898003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7ff6fe9dd640 10 monclient: shutdown 2024-09-15T02:43:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7ff6fe9dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e4004780 msgr2=0x7ff6e4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e4004780 0x7ff6e4004bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4005500 tx=0x7ff6f40090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fafdcc57640 1 -- 172.21.15.53:0/1402669208 wait complete. 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fafdcc57640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fafdcc57640 10 monclient: monmap: 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fafdcc57640 1 librados: starting msgr at 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fafdcc57640 1 librados: starting objecter 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fafdcc57640 1 Processor -- start 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/665032328 shutdown_connections 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/665032328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e4004780 0x7ff6e4004bf0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4005500 tx=0x7ff6f40090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/665032328 >> 172.21.15.53:0/665032328 conn(0x7ff6f0039cd0 msgr2=0x7ff6f0035ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7e40a4f640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fd8a3fff640 10 monclient(hunting): picked mon.a con 0x7fd898003a50 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fd8a3fff640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd898004780 0x7fd898004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7e40a4f640 10 monclient: monmap: 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.101 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.017+0000 7fa4b8109640 1 -- 172.21.15.53:0/4227499138 wait complete. 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fa4b8109640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7fa4b8109640 10 monclient: monmap: 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fa4b8109640 1 librados: starting msgr at 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fa4b8109640 1 librados: starting objecter 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fa4b8109640 1 Processor -- start 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 1 -- 172.21.15.53:0/1852455683 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8802ad40 con 0x7f7a84008b70 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: dump: 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 1 -- 172.21.15.53:0/1852455683 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a88032440 con 0x7f7a84008b70 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 1 -- 172.21.15.53:0/1852455683 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8802aea0 con 0x7f7a84008b70 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a911e9640 10 monclient: dump: 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a931ed640 5 monclient: authenticate success, global_id 4342 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a931ed640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7a931ed640 4 set_mon_vals no callback set 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f6a6399d640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f6a6399d640 10 monclient: monmap: 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7e40a4f640 1 librados: starting msgr at 2024-09-15T02:43:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7e40a4f640 1 librados: starting objecter 2024-09-15T02:43:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fd8a3fff640 10 monclient(hunting): picked mon.b con 0x7fd898004780 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd898004780 0x7fd898004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fd8b160b640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd898003a50 0x7fd898003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f6a6399d640 1 librados: starting msgr at 2024-09-15T02:43:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f6a6399d640 1 librados: starting objecter 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fafdcc57640 1 -- start start 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fafdcc57640 1 librados: setting wanted keys 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fafdcc57640 1 librados: calling monclient init 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fafdcc57640 10 monclient: init 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafd806a8c0 0x7fafd80a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): picked mon.noname-c con 0x7fafd806a8c0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd81398b0 0x7fafd80a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): picked mon.noname-b con 0x7fafd81398b0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafd813a560 0x7fafd8154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): picked mon.noname-a con 0x7fafd813a560 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8154b30 con 0x7fafd813a560 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80aa6a0 con 0x7fafd81398b0 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80a6550 con 0x7fafd806a8c0 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafdcc57640 10 monclient(hunting): authenticate will time out at 741.590129s 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7e40a4f640 1 Processor -- start 2024-09-15T02:43:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f7e40a4f640 1 -- start start 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f7e40a4f640 1 librados: setting wanted keys 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f7e40a4f640 1 librados: calling monclient init 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f7e40a4f640 10 monclient: init 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fd8a3fff640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 0x7fd898005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fd8a3fff640 10 monclient(hunting): picked mon.c con 0x7fd8ac145340 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac142740 con 0x7fd898003a50 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0b0e70 con 0x7fd898004780 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b1e0c640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 0x7fd898005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89800b9d0 con 0x7fd8ac145340 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8a3fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b0e0a640 10 monclient(hunting): get_auth_request con 0x7fd898004780 auth_method 0 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b0e0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b0e0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b0e0a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7f67e9f5d640 1 -- start start 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.018+0000 7f67e9f5d640 10 monclient: init 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f67e413ee10 0x7f67e40ab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f67e413ee10 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e40abb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f67e413f1e0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67e40ad1a0 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e40acd10 con 0x7f67e40ad1a0 2024-09-15T02:43:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e406efa0 con 0x7f67e413ee10 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e406a970 con 0x7f67e413f1e0 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e9f5d640 10 monclient(hunting): authenticate will time out at 741.589507s 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e40abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e40abb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47172/0 (socket says 172.21.15.53:47172) 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 1 -- 172.21.15.53:0/95076804 learned_addr learned my addr 172.21.15.53:0/95076804 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 10 monclient(hunting): get_auth_request con 0x7f67e413f1e0 auth_method 0 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7f67e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e37fe640 1 --2- 172.21.15.53:0/95076804 >> v2:172.21.15.104:3300/0 conn(0x7f67e413ee10 0x7f67e40ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 1 -- 172.21.15.53:0/95076804 >> v2:172.21.15.104:3300/0 conn(0x7f67e413ee10 msgr2=0x7f67e40ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e27fc640 1 -- 172.21.15.53:0/95076804 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728563902 0 0) 0x7f67e40acd10 con 0x7f67e40ad1a0 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 1 --2- 172.21.15.53:0/95076804 >> v2:172.21.15.104:3300/0 conn(0x7f67e413ee10 0x7f67e40ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 1 -- 172.21.15.53:0/95076804 >> v1:172.21.15.104:6789/0 conn(0x7f67e40ad1a0 legacy=0x7f67e40ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f6a6399d640 1 Processor -- start 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafd813a560 0x7fafd8154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafd813a560 0x7fafd8154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51794/0 (socket says 172.21.15.53:51794) 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 1 -- 172.21.15.53:0/141445028 learned_addr learned my addr 172.21.15.53:0/141445028 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): get_auth_request con 0x7fafd813a560 auth_method 0 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fa4b8109640 1 -- start start 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fa4b8109640 1 librados: setting wanted keys 2024-09-15T02:43:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fa4b8109640 1 librados: calling monclient init 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.019+0000 7fa4b8109640 10 monclient: init 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa4b00a98d0 0x7fa4b00a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b00a98d0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4b00a9ca0 0x7fa4b00a7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b00a9ca0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 0x7fa4b00a9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): picked mon.noname-b con 0x7fa4b0141300 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f6a6399d640 1 -- start start 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00a6e00 con 0x7fa4b00a9ca0 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00a5500 con 0x7fa4b0141300 2024-09-15T02:43:25.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00a9680 con 0x7fa4b00a98d0 2024-09-15T02:43:25.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b8109640 10 monclient(hunting): authenticate will time out at 741.590400s 2024-09-15T02:43:25.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f6a6399d640 1 librados: setting wanted keys 2024-09-15T02:43:25.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f6a6399d640 1 librados: calling monclient init 2024-09-15T02:43:25.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa4b00a98d0 0x7fa4b00a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f6a6399d640 10 monclient: init 2024-09-15T02:43:25.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa4b00a98d0 0x7fa4b00a6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47188/0 (socket says 172.21.15.53:47188) 2024-09-15T02:43:25.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 1 -- 172.21.15.53:0/2263763715 learned_addr learned my addr 172.21.15.53:0/2263763715 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b160b640 10 monclient(hunting): get_auth_request con 0x7fd898003a50 auth_method 0 2024-09-15T02:43:25.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b160b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b160b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:25.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-15T02:43:25.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd6d9d640 1 -- 172.21.15.53:0/141445028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafd806a8c0 msgr2=0x7fafd80a8f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:43:25.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b1e0c640 10 monclient(hunting): get_auth_request con 0x7fd8ac145340 auth_method 0 2024-09-15T02:43:25.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b1e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b1e0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b1e0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:43:25.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fafd5d9b640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd81398b0 0x7fafd80a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafd806a8c0 0x7fafd80a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fd8b386f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 1 -- 172.21.15.53:0/141445028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd81398b0 msgr2=0x7fafd80a9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:43:25.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e3c06d090 0x7f7e3c14c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3c06d090 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:25.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06d700 0x7f7e3c14cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b1e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b1e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b1e0c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:25.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a5c0abcb0 0x7f6a5c13fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a5c0abcb0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a5c0b0870 0x7f6a5c150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a5c0b0870 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0b0c40 0x7f6a5c06dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a5c0b0c40 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c06e000 con 0x7f6a5c0abcb0 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c118de0 con 0x7f6a5c0b0c40 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c06e220 con 0x7f6a5c0b0870 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f6a6399d640 10 monclient(hunting): authenticate will time out at 741.591840s 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0b0c40 0x7f6a5c06dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0b0c40 0x7f6a5c06dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60364/0 (socket says 172.21.15.53:60364) 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 1 -- 172.21.15.53:0/989860089 learned_addr learned my addr 172.21.15.53:0/989860089 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 10 monclient(hunting): get_auth_request con 0x7f6a5c0b0c40 auth_method 0 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a5c0b0870 0x7f6a5c150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): get_auth_request con 0x7f6a5c0b0870 auth_method 0 2024-09-15T02:43:25.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a61f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/665032328 shutdown_connections 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/665032328 wait complete. 2024-09-15T02:43:25.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7ff6fe9dd640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7ff6fe9dd640 10 monclient: monmap: 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.129 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd81398b0 0x7fafd80a9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 1 monclient: found mon.noname-a 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 1 -- 172.21.15.53:0/141445028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd000ab30 con 0x7fafd813a560 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd6d9d640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafd813a560 0x7fafd8154f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafd000ab00 tx=0x7fafd0039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd5d9b640 10 monclient: get_auth_request con 0x7fafd81398b0 auth_method 0 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafd5d9b640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd81398b0 0x7fafd80a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafce7fc640 10 client.?.objecter ms_handle_connect 0x7fafd813a560 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fafce7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafce7fc640 1 -- 172.21.15.53:0/141445028 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0032af0 con 0x7fafd813a560 2024-09-15T02:43:25.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafce7fc640 10 monclient: dump: 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.131 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafce7fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafdcc57640 5 monclient: authenticate success, global_id 4350 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafdcc57640 10 monclient: _renew_subs 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafdcc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd80a6710 con 0x7fafd813a560 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 10 monclient(hunting): get_auth_request con 0x7fa4b00a98d0 auth_method 0 2024-09-15T02:43:25.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7fa4b5ea5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b56a4640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4b00a9ca0 0x7fa4b00a7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 1 -- 172.21.15.53:0/2263763715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 msgr2=0x7fa4b00a9140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 0x7fa4b00a9140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 1 -- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4b00a9ca0 msgr2=0x7fa4b00a7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4b00a9ca0 0x7fa4b00a7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 1 -- 172.21.15.53:0/2263763715 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac009390 con 0x7fa4b00a98d0 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b56a4640 10 monclient: get_auth_request con 0x7fa4b00a9ca0 auth_method 0 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b56a4640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4b00a9ca0 0x7fa4b00a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4b5ea5640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa4b00a98d0 0x7fa4b00a6230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800cec0 tx=0x7fa4a8001c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa4b00a98d0 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fa4a5ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e3a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e3c06d090 0x7f7e3c14c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e3a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e3c06d090 0x7f7e3c14c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:60362/0 (socket says 172.21.15.53:60362) 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f7e3c06d700 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e3a59c640 1 -- 172.21.15.53:0/1959147151 learned_addr learned my addr 172.21.15.53:0/1959147151 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06dad0 0x7f7e3c133cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e39d9b640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06d700 0x7f7e3c14cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3c06dad0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c133820 con 0x7f7e3c06dad0 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c13fac0 con 0x7f7e3c06d090 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c13ee90 con 0x7f7e3c06d700 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e40a4f640 10 monclient(hunting): authenticate will time out at 741.591866s 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e3a59c640 10 monclient(hunting): get_auth_request con 0x7f7e3c06d090 auth_method 0 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e39d9b640 10 monclient(hunting): get_auth_request con 0x7f7e3c06d700 auth_method 0 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7e39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7e39d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7e39d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7e39d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_done global_id 4348 payload 293 2024-09-15T02:43:25.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 1 -- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 msgr2=0x7fd898005130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac145340 0x7fd898005130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 1 -- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd898003a50 msgr2=0x7fd898003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd898003a50 0x7fd898003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 1 monclient: found mon.b 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 1 -- 172.21.15.53:0/1930949188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8980034e0 con 0x7fd898004780 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7fd8b0e0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fd8b160b640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd898003a50 0x7fd898003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7a931ed640 10 monclient: shutdown 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7a931ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7a931ed640 1 -- 172.21.15.53:0/1852455683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008b70 msgr2=0x7f7a84036120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7a931ed640 1 --2- 172.21.15.53:0/1852455683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008b70 0x7f7a84036120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a84032320 tx=0x7f7a880091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7a931ed640 1 -- 172.21.15.53:0/1852455683 shutdown_connections 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f7a931ed640 1 --2- 172.21.15.53:0/1852455683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008b70 0x7f7a84036120 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 1 --2- 172.21.15.53:0/1852455683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a84008740 0x7f7a84035be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 1 --2- 172.21.15.53:0/1852455683 >> v2:172.21.15.53:3300/0 conn(0x7f7a840073b0 0x7f7a84032c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 1 -- 172.21.15.53:0/1852455683 >> 172.21.15.53:0/1852455683 conn(0x7f7a84037ce0 msgr2=0x7f7a840380e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 1 -- 172.21.15.53:0/1852455683 shutdown_connections 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 1 -- 172.21.15.53:0/1852455683 wait complete. 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 10 monclient: monmap: 2024-09-15T02:43:25.138 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 1 librados: starting msgr at 2024-09-15T02:43:25.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f7a931ed640 1 librados: starting objecter 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 1 monclient: found mon.noname-b 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.020+0000 7f67e2ffd640 1 -- 172.21.15.53:0/95076804 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e40add50 con 0x7f67e413f1e0 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f67e27fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f67e37fe640 10 monclient: get_auth_request con 0x7f67e413ee10 auth_method 0 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f67e37fe640 1 --2- 172.21.15.53:0/95076804 >> v2:172.21.15.104:3300/0 conn(0x7f67e413ee10 0x7f67e40ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.021+0000 7f67e2ffd640 1 --2- 172.21.15.53:0/95076804 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e40abb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009b70 tx=0x7f67d40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9d39b27ff9423f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 1 -- 172.21.15.53:0/95076804 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4031400 con 0x7f67e413f1e0 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:25.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: dump: 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e9f5d640 5 monclient: authenticate success, global_id 4298 2024-09-15T02:43:25.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e9f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 1 -- 172.21.15.53:0/95076804 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d402d050 con 0x7f67e413f1e0 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 1 -- 172.21.15.53:0/95076804 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4031790 con 0x7f67e413f1e0 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e27fc640 10 monclient: dump: 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.142 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.143 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.143 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7ff6fe9dd640 1 librados: starting msgr at 2024-09-15T02:43:25.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7ff6fe9dd640 1 librados: starting objecter 2024-09-15T02:43:25.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7ff6fe9dd640 1 Processor -- start 2024-09-15T02:43:25.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd898004780 0x7fd898004bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009b70 tx=0x7fd89c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda614a54e957810 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e9f5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f67e9f5d640 4 set_mon_vals no callback set 2024-09-15T02:43:25.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f67e9f5d640 10 monclient: shutdown 2024-09-15T02:43:25.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f67e9f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f67e9f5d640 1 -- 172.21.15.53:0/95076804 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 msgr2=0x7f67e40abb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/95076804 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e40abb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009b70 tx=0x7f67d40091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 1 -- 172.21.15.53:0/2263763715 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800a980 con 0x7fa4b00a98d0 2024-09-15T02:43:25.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 10 monclient: dump: 2024-09-15T02:43:25.145 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.145 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.145 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 1 -- 172.21.15.53:0/2263763715 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a800aae0 con 0x7fa4b00a98d0 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7fa4a7fff640 4 set_mon_vals no callback set 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4a5ffb640 1 -- 172.21.15.53:0/2263763715 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800ae60 con 0x7fa4b00a98d0 2024-09-15T02:43:25.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4a5ffb640 10 monclient: dump: 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4b8109640 5 monclient: authenticate success, global_id 4301 2024-09-15T02:43:25.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4b8109640 10 monclient: _renew_subs 2024-09-15T02:43:25.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c031400 con 0x7fd898004780 2024-09-15T02:43:25.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f67e9f5d640 1 -- 172.21.15.53:0/95076804 shutdown_connections 2024-09-15T02:43:25.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/95076804 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e40abb60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/95076804 >> v2:172.21.15.104:3300/0 conn(0x7f67e413ee10 0x7f67e40ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f67e9f5d640 1 -- 172.21.15.53:0/95076804 >> 172.21.15.53:0/95076804 conn(0x7f67e4063810 msgr2=0x7f67e40b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7e39d9b640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7e39d9b640 1 -- 172.21.15.53:0/1959147151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e3c06d090 msgr2=0x7f7e3c14c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7e39d9b640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e3c06d090 0x7f7e3c14c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 1 -- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06dad0 msgr2=0x7f7e3c133cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06dad0 0x7f7e3c133cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c009390 con 0x7f7e3c06d700 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7ff6fe9dd640 1 -- start start 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7ff6fe9dd640 1 librados: setting wanted keys 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7ff6fe9dd640 1 librados: calling monclient init 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7ff6fe9dd640 10 monclient: init 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6f00bb4a0 0x7ff6f002ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f00bb4a0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6f002d4f0 0x7ff6f00e6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f002d4f0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6f00e6680 0x7ff6f00d6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f00e6680 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00bb870 con 0x7ff6f00e6680 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00e6af0 con 0x7ff6f002d4f0 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f002edd0 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe9dd640 10 monclient(hunting): authenticate will time out at 741.594470s 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe1dc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6f00e6680 0x7ff6f00d6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe1dc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6f00e6680 0x7ff6f00d6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51844/0 (socket says 172.21.15.53:51844) 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe1dc640 1 -- 172.21.15.53:0/3582228471 learned_addr learned my addr 172.21.15.53:0/3582228471 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe1dc640 10 monclient(hunting): get_auth_request con 0x7ff6f00e6680 auth_method 0 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe1dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe1dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fe1dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fd9db640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6f00bb4a0 0x7ff6f002ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fd9db640 10 monclient(hunting): get_auth_request con 0x7ff6f00bb4a0 auth_method 0 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fd9db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fd9db640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fd9db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fd9db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fafdcc57640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fafdcc57640 10 monclient: _renew_subs 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fafdcc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd80aa3a0 con 0x7fafd813a560 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fafdcc57640 1 librados: init done 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fafce7fc640 1 -- 172.21.15.53:0/141445028 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd0032c50 con 0x7fafd813a560 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fafce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fafdcc57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fafd4d99640 4 set_mon_vals no callback set 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4b8109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b00a8480 con 0x7fa4b00a98d0 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fa4b8109640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fa4b8109640 10 monclient: _renew_subs 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fa4b8109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b006d470 con 0x7fa4b00a98d0 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fa4b8109640 1 librados: init done 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e39d9b640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06d700 0x7f7e3c14cb50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c810 tx=0x7f7e30007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e2a7fc640 10 client.?.objecter ms_handle_connect 0x7f7e3c06d700 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7e2a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 10 monclient: dump: 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c02d050 con 0x7fd898004780 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 1 -- 172.21.15.53:0/1930949188 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c031790 con 0x7fd898004780 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8a3fff640 10 monclient: dump: 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.156 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8b386f640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7fd8b386f640 4 set_mon_vals no callback set 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7a931ed640 1 Processor -- start 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7a931ed640 1 -- start start 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7a931ed640 1 librados: setting wanted keys 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7a931ed640 1 librados: calling monclient init 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f7a931ed640 10 monclient: init 2024-09-15T02:43:25.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a840073b0 0x7f7a8402ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): picked mon.noname-c con 0x7f7a840073b0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008740 0x7f7a8402d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): picked mon.noname-b con 0x7f7a84008740 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a84008b70 0x7f7a840be940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): picked mon.noname-a con 0x7f7a84008b70 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8402f440 con 0x7f7a84008b70 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840c0a40 con 0x7f7a84008740 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840c1850 con 0x7f7a840073b0 2024-09-15T02:43:25.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a931ed640 10 monclient(hunting): authenticate will time out at 741.594192s 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a84008b70 0x7f7a840be940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a84008b70 0x7f7a840be940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51836/0 (socket says 172.21.15.53:51836) 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 1 -- 172.21.15.53:0/4114574554 learned_addr learned my addr 172.21.15.53:0/4114574554 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 10 monclient(hunting): get_auth_request con 0x7f7a84008b70 auth_method 0 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7ff6fd9db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.024+0000 7f7a929ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-09-15T02:43:25.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 1 -- 172.21.15.53:0/4114574554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a840073b0 msgr2=0x7f7a8402ed70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:43:25.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 1 --2- 172.21.15.53:0/4114574554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a840073b0 0x7f7a8402ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-09-15T02:43:25.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 1 -- 172.21.15.53:0/4114574554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008740 msgr2=0x7f7a8402d560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:43:25.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 1 -- 172.21.15.53:0/3582228471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6f002d4f0 msgr2=0x7ff6f00e6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:43:25.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6f002d4f0 0x7ff6f00e6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 1 --2- 172.21.15.53:0/4114574554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008740 0x7f7a8402d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 1 -- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6f00e6680 msgr2=0x7ff6f00d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6f00e6680 0x7ff6f00d6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 1 monclient: found mon.noname-a 2024-09-15T02:43:25.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafce7fc640 1 -- 172.21.15.53:0/141445028 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0045460 con 0x7fafd813a560 2024-09-15T02:43:25.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafdcc57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 1 -- 172.21.15.53:0/4114574554 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a88009450 con 0x7f7a84008b70 2024-09-15T02:43:25.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7ff6fd9db640 1 -- 172.21.15.53:0/3582228471 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f002d980 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafdcc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd8154cb0 con 0x7fafd813a560 2024-09-15T02:43:25.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:25.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafce7fc640 10 monclient: dump: 2024-09-15T02:43:25.165 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.165 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fafce7fc640 1 -- 172.21.15.53:0/141445028 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fafd00457f0 con 0x7fafd813a560 2024-09-15T02:43:25.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fa4b8109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a827fc640 10 client.?.objecter ms_handle_connect 0x7f7a84008b70 2024-09-15T02:43:25.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a827fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f7a929ec640 1 --2- 172.21.15.53:0/4114574554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a84008b70 0x7f7a840be940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009210 tx=0x7f7a88032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f67e9f5d640 1 -- 172.21.15.53:0/95076804 shutdown_connections 2024-09-15T02:43:25.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fa4b8109640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7fa4b8109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f67e9f5d640 1 -- 172.21.15.53:0/95076804 wait complete. 2024-09-15T02:43:25.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b00646e0 con 0x7fa4b00a98d0 2024-09-15T02:43:25.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-15T02:43:25.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 1 -- 172.21.15.53:0/989860089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0b0c40 msgr2=0x7f6a5c06dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0b0c40 0x7f6a5c06dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 1 -- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a5c0abcb0 msgr2=0x7f6a5c13fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a5c0abcb0 0x7f6a5c13fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.022+0000 7f6a60f38640 1 -- 172.21.15.53:0/989860089 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a54009a20 con 0x7f6a5c0b0870 2024-09-15T02:43:25.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a60f38640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a5c0b0870 0x7f6a5c150990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00b4c0 tx=0x7f6a4c006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 client.?.objecter ms_handle_connect 0x7f6a5c0b0870 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 1 -- 172.21.15.53:0/989860089 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c011630 con 0x7f6a5c0b0870 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: dump: 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.170 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 1 -- 172.21.15.53:0/989860089 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c0117d0 con 0x7f6a5c0b0870 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 1 -- 172.21.15.53:0/989860089 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c011b90 con 0x7f6a5c0b0870 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a517fa640 10 monclient: dump: 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.171 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 5 monclient: authenticate success, global_id 4307 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 10 monclient: _renew_subs 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5c138c40 con 0x7f6a5c0b0870 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 10 monclient: _renew_subs 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5c06e980 con 0x7f6a5c0b0870 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7f6a6399d640 1 librados: init done 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7fafce7fc640 1 --2- 172.21.15.53:0/141445028 >> v2:172.21.15.104:6816/32804 conn(0x7fafc006e8d0 0x7fafc0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fafce7fc640 1 -- 172.21.15.53:0/141445028 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7fafd00ad010 con 0x7fafd813a560 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fafce7fc640 10 client.4350.objecter ms_dispatch 0x7fafd8000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fafce7fc640 3 client.4350.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fafce7fc640 3 client.4350.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fafce7fc640 20 client.4350.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 1 -- 172.21.15.53:0/1959147151 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30005870 con 0x7f7e3c06d700 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: dump: 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7f67e9f5d640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7f67e9f5d640 10 monclient: monmap: 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.175 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.023+0000 7f6a52ffd640 4 set_mon_vals no callback set 2024-09-15T02:43:25.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 1 -- 172.21.15.53:0/1959147151 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e300059d0 con 0x7f7e3c06d700 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 1 -- 172.21.15.53:0/1959147151 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30005d90 con 0x7f7e3c06d700 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e2a7fc640 10 monclient: dump: 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.177 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e38d99640 4 set_mon_vals no callback set 2024-09-15T02:43:25.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f67e9f5d640 1 librados: starting msgr at 2024-09-15T02:43:25.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f67e9f5d640 1 librados: starting objecter 2024-09-15T02:43:25.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f67e9f5d640 1 Processor -- start 2024-09-15T02:43:25.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e40a4f640 5 monclient: authenticate success, global_id 4304 2024-09-15T02:43:25.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e40a4f640 10 monclient: _renew_subs 2024-09-15T02:43:25.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e40a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fafd659c640 1 --2- 172.21.15.53:0/141445028 >> v2:172.21.15.104:6816/32804 conn(0x7fafc006e8d0 0x7fafc0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c14d090 con 0x7f7e3c06d700 2024-09-15T02:43:25.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fd8b386f640 10 monclient: shutdown 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fd8b386f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fd8b386f640 1 -- 172.21.15.53:0/1930949188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd898004780 msgr2=0x7fd898004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fd8b386f640 1 --2- 172.21.15.53:0/1930949188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd898004780 0x7fd898004bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009b70 tx=0x7fd89c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fd8b1e0c640 1 -- 172.21.15.53:0/1930949188 reap_dead start 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fd8b386f640 1 -- 172.21.15.53:0/1930949188 shutdown_connections 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.025+0000 7fd8b386f640 1 -- 172.21.15.53:0/1930949188 >> 172.21.15.53:0/1930949188 conn(0x7fd8ac067b00 msgr2=0x7fd8ac0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fd8b386f640 1 -- 172.21.15.53:0/1930949188 shutdown_connections 2024-09-15T02:43:25.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7fafd659c640 10 monclient: get_auth_request con 0x7fafc006e8d0 auth_method 0 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7ff6edffb640 10 client.?.objecter ms_handle_connect 0x7ff6f00bb4a0 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7ff6edffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.026+0000 7ff6fd9db640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6f00bb4a0 0x7ff6f002ca20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0003d90 tx=0x7ff6e8009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 1 -- 172.21.15.53:0/3582228471 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8012590 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: dump: 2024-09-15T02:43:25.181 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 1 -- 172.21.15.53:0/3582228471 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e80126f0 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 1 -- 172.21.15.53:0/3582228471 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8012590 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6edffb640 10 monclient: dump: 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6effff640 4 set_mon_vals no callback set 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6fe9dd640 5 monclient: authenticate success, global_id 4310 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6fe9dd640 10 monclient: _renew_subs 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7ff6fe9dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f002dea0 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7ff6fe9dd640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7ff6fe9dd640 10 monclient: _renew_subs 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7ff6fe9dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f00d7420 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7ff6fe9dd640 1 librados: init done 2024-09-15T02:43:25.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f6a517fa640 1 -- 172.21.15.53:0/989860089 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f6a4c0193b0 con 0x7f6a5c0b0870 2024-09-15T02:43:25.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f67e9f5d640 1 -- start start 2024-09-15T02:43:25.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f67e9f5d640 1 librados: setting wanted keys 2024-09-15T02:43:25.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f67e9f5d640 1 librados: calling monclient init 2024-09-15T02:43:25.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f7e40a4f640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 1 -- 172.21.15.53:0/4114574554 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88031a60 con 0x7f7a84008b70 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: dump: 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.186 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 1 -- 172.21.15.53:0/4114574554 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a88031bc0 con 0x7f7a84008b70 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 1 -- 172.21.15.53:0/4114574554 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8803c440 con 0x7f7a84008b70 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a827fc640 10 monclient: dump: 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.187 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 5 monclient: authenticate success, global_id 4353 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 10 monclient: _renew_subs 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8402daa0 con 0x7f7a84008b70 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 10 monclient: _renew_subs 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8402e2e0 con 0x7f7a84008b70 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a931ed640 1 librados: init done 2024-09-15T02:43:25.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f7a83fff640 4 set_mon_vals no callback set 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.027+0000 7f67e9f5d640 10 monclient: init 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f67e9f5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f67e9f5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f67e9f5d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67e40ad1a0 0x7f67e40b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f67e9f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f67e40ad1a0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f67e9f5d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67e413ee10 0x7f67e4134100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f67e9f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f67e413ee10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f67e9f5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e4134640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f7e40a4f640 10 monclient: _renew_subs 2024-09-15T02:43:25.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f7e40a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c134390 con 0x7f7e3c06d700 2024-09-15T02:43:25.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.028+0000 7f6a517fa640 1 --2- 172.21.15.53:0/989860089 >> v2:172.21.15.104:6816/32804 conn(0x7f6a4006e8d0 0x7f6a40070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f7e40a4f640 1 librados: init done 2024-09-15T02:43:25.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f67e413f1e0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e4134640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67e40ad1a0 0x7f67e40b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e4134640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47226/0 (socket says 172.21.15.53:47226) 2024-09-15T02:43:25.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67e40ad1a0 0x7f67e40b1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51860/0 (socket says 172.21.15.53:51860) 2024-09-15T02:43:25.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 -- 172.21.15.53:0/1258558762 learned_addr learned my addr 172.21.15.53:0/1258558762 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e2ffd640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67e413ee10 0x7f67e4134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e413a7e0 con 0x7f67e40ad1a0 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e4139680 con 0x7f67e413ee10 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e413aa50 con 0x7f67e413f1e0 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e9f5d640 10 monclient(hunting): authenticate will time out at 741.599471s 2024-09-15T02:43:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7fa4a5ffb640 1 -- 172.21.15.53:0/2263763715 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fa4a8007300 con 0x7fa4b00a98d0 2024-09-15T02:43:25.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f6a6399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f7a931ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f7a931ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f7a931ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f6a6399d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7ff6fe9dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7ff6fe9dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7ff6fe9dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f002ef60 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a74005010 con 0x7f7a84008b70 2024-09-15T02:43:25.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f6a6399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fafd659c640 1 --2- 172.21.15.53:0/141445028 >> v2:172.21.15.104:6816/32804 conn(0x7fafc006e8d0 0x7fafc0070d80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fafc80016a0 tx=0x7fafc8007ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fafce7fc640 10 client.4350.objecter ms_handle_connect 0x7fafc006e8d0 2024-09-15T02:43:25.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4a5ffb640 1 --2- 172.21.15.53:0/2263763715 >> v2:172.21.15.104:6816/32804 conn(0x7fa49406e8d0 0x7fa494070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5c06e480 con 0x7f6a5c0b0870 2024-09-15T02:43:25.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f6a61739640 1 --2- 172.21.15.53:0/989860089 >> v2:172.21.15.104:6816/32804 conn(0x7f6a4006e8d0 0x7f6a40070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f6a61739640 10 monclient: get_auth_request con 0x7f6a4006e8d0 auth_method 0 2024-09-15T02:43:25.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): get_auth_request con 0x7f67e413f1e0 auth_method 0 2024-09-15T02:43:25.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 10 monclient(hunting): get_auth_request con 0x7f67e40ad1a0 auth_method 0 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 -- 172.21.15.53:0/1258558762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67e413ee10 msgr2=0x7f67e4134100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67e413ee10 0x7f67e4134100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 -- 172.21.15.53:0/1258558762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67e40ad1a0 msgr2=0x7f67e40b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67e40ad1a0 0x7f67e40b1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.029+0000 7f67e3fff640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d4009450 con 0x7f67e413f1e0 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f67e3fff640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e4134640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00c840 tx=0x7f67dc0015f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4a5ffb640 1 -- 172.21.15.53:0/2263763715 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7fa4a808b9b0 con 0x7fa4b00a98d0 2024-09-15T02:43:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4a5ffb640 10 client.4301.objecter ms_dispatch 0x7fa4b0000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fd8b386f640 1 -- 172.21.15.53:0/1930949188 wait complete. 2024-09-15T02:43:25.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4a5ffb640 3 client.4301.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f67e2ffd640 10 monclient: get_auth_request con 0x7f67e413ee10 auth_method 0 2024-09-15T02:43:25.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f67e2ffd640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67e413ee10 0x7f67e4134100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:43:25.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4a5ffb640 3 client.4301.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4b56a4640 1 --2- 172.21.15.53:0/2263763715 >> v2:172.21.15.104:6816/32804 conn(0x7fa49406e8d0 0x7fa494070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f67cb7fe640 10 client.?.objecter ms_handle_connect 0x7f67e413f1e0 2024-09-15T02:43:25.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f67cb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4b56a4640 10 monclient: get_auth_request con 0x7fa49406e8d0 auth_method 0 2024-09-15T02:43:25.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f6a517fa640 1 -- 172.21.15.53:0/989860089 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f6a4c059050 con 0x7f6a5c0b0870 2024-09-15T02:43:25.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7fa4a5ffb640 20 client.4301.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f6a517fa640 10 client.4307.objecter ms_dispatch 0x7f6a5c000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f6a517fa640 3 client.4307.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f6a517fa640 3 client.4307.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.030+0000 7f6a517fa640 20 client.4307.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7f6a61739640 1 --2- 172.21.15.53:0/989860089 >> v2:172.21.15.104:6816/32804 conn(0x7f6a4006e8d0 0x7f6a40070d80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009740 tx=0x7f6a58006c50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7f6a517fa640 10 client.4307.objecter ms_handle_connect 0x7f6a4006e8d0 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7fa4a5ffb640 10 client.4301.objecter ms_handle_connect 0x7fa49406e8d0 2024-09-15T02:43:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7fa4b56a4640 1 --2- 172.21.15.53:0/2263763715 >> v2:172.21.15.104:6816/32804 conn(0x7fa49406e8d0 0x7fa494070d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4b006ec40 tx=0x7fa4a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7fd8b386f640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7f7e40a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7f7e40a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7f7e40a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7fd8b386f640 10 monclient: monmap: 2024-09-15T02:43:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c063ff0 con 0x7f7e3c06d700 2024-09-15T02:43:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.031+0000 7fd8b386f640 1 librados: starting msgr at 2024-09-15T02:43:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.032+0000 7fd8b386f640 1 librados: starting objecter 2024-09-15T02:43:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.032+0000 7fd8b386f640 1 Processor -- start 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.032+0000 7fd8b386f640 1 -- start start 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.032+0000 7fd8b386f640 1 librados: setting wanted keys 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.032+0000 7fd8b386f640 1 librados: calling monclient init 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.032+0000 7fd8b386f640 10 monclient: init 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac154e60 0x7fd8ac06cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 10 monclient(hunting): picked mon.noname-c con 0x7fd8ac154e60 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8ac06d270 0x7fd8ac0a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 10 monclient(hunting): picked mon.noname-a con 0x7fd8ac06d270 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ac0a5a00 0x7fd8ac0a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b160b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac154e60 0x7fd8ac06cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b160b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac154e60 0x7fd8ac06cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47230/0 (socket says 172.21.15.53:47230) 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b0e0a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8ac06d270 0x7fd8ac0a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b386f640 10 monclient(hunting): picked mon.noname-b con 0x7fd8ac0a5a00 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.033+0000 7fd8b160b640 1 -- 172.21.15.53:0/2210446254 learned_addr learned my addr 172.21.15.53:0/2210446254 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0a7090 con 0x7fd8ac06d270 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ac0a5a00 0x7fd8ac0a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0a6390 con 0x7fd8ac0a5a00 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac1552b0 con 0x7fd8ac154e60 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b386f640 10 monclient(hunting): authenticate will time out at 741.604235s 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): get_auth_request con 0x7fd8ac06d270 auth_method 0 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b160b640 10 monclient(hunting): get_auth_request con 0x7fd8ac154e60 auth_method 0 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b160b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b160b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fa4a5ffb640 1 -- 172.21.15.53:0/2263763715 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa4a8056040 con 0x7fa4b00a98d0 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fa4a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fa4a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 10 monclient(hunting): get_auth_request con 0x7fd8ac0a5a00 auth_method 0 2024-09-15T02:43:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b1e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b160b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 1 -- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac154e60 msgr2=0x7fd8ac06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac154e60 0x7fd8ac06cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 1 -- 172.21.15.53:0/2210446254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ac0a5a00 msgr2=0x7fd8ac0a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ac0a5a00 0x7fd8ac0a5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 1 monclient: found mon.noname-a 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.034+0000 7fd8b0e0a640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac06d760 con 0x7fd8ac06d270 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fd8b1e0c640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ac0a5a00 0x7fd8ac0a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fd8b160b640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac154e60 0x7fd8ac06cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:25.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fafce7fc640 1 -- 172.21.15.53:0/141445028 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fafd00dee40 con 0x7fafd813a560 2024-09-15T02:43:25.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fafce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fafce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fd8b0e0a640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8ac06d270 0x7fd8ac0a54c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd89c031000 tx=0x7fd89c035000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fd8a17fa640 10 client.?.objecter ms_handle_connect 0x7fd8ac06d270 2024-09-15T02:43:25.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.035+0000 7fd8a17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.036+0000 7f7e2a7fc640 1 -- 172.21.15.53:0/1959147151 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f7e30007300 con 0x7f7e3c06d700 2024-09-15T02:43:25.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.036+0000 7f7e2a7fc640 1 --2- 172.21.15.53:0/1959147151 >> v2:172.21.15.104:6816/32804 conn(0x7f7e1c06e8d0 0x7f7e1c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.036+0000 7f7a827fc640 1 -- 172.21.15.53:0/4114574554 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f7a8803c830 con 0x7f7a84008b70 2024-09-15T02:43:25.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7e3a59c640 1 --2- 172.21.15.53:0/1959147151 >> v2:172.21.15.104:6816/32804 conn(0x7f7e1c06e8d0 0x7f7e1c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7e3a59c640 10 monclient: get_auth_request con 0x7f7e1c06e8d0 auth_method 0 2024-09-15T02:43:25.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a827fc640 1 --2- 172.21.15.53:0/4114574554 >> v2:172.21.15.104:6816/32804 conn(0x7f7a7006e8d0 0x7f7a70070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7e2a7fc640 10 client.4304.objecter ms_handle_connect 0x7f7e1c06e8d0 2024-09-15T02:43:25.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7e3a59c640 1 --2- 172.21.15.53:0/1959147151 >> v2:172.21.15.104:6816/32804 conn(0x7f7e1c06e8d0 0x7f7e1c070d80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c039040 tx=0x7f7e2c036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a827fc640 1 -- 172.21.15.53:0/4114574554 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f7a880acd00 con 0x7f7a84008b70 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a827fc640 10 client.4353.objecter ms_dispatch 0x7f7a8c000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a827fc640 3 client.4353.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a827fc640 3 client.4353.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a921eb640 1 --2- 172.21.15.53:0/4114574554 >> v2:172.21.15.104:6816/32804 conn(0x7f7a7006e8d0 0x7f7a70070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a921eb640 10 monclient: get_auth_request con 0x7f7a7006e8d0 auth_method 0 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.037+0000 7f7a827fc640 20 client.4353.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.038+0000 7f7a827fc640 10 client.4353.objecter ms_handle_connect 0x7f7a7006e8d0 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.038+0000 7f7a921eb640 1 --2- 172.21.15.53:0/4114574554 >> v2:172.21.15.104:6816/32804 conn(0x7f7a7006e8d0 0x7f7a70070d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c066cc0 tx=0x7f7a8c064d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f67cb7fe640 1 -- 172.21.15.53:0/1258558762 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc00ce30 con 0x7f67e413f1e0 2024-09-15T02:43:25.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f6a517fa640 1 -- 172.21.15.53:0/989860089 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6a5c06e480 con 0x7f6a5c0b0870 2024-09-15T02:43:25.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f6a517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f6a517fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f67cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f67cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f7e2a7fc640 1 -- 172.21.15.53:0/1959147151 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f7e30059f60 con 0x7f7e3c06d700 2024-09-15T02:43:25.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f7e2a7fc640 10 client.4304.objecter ms_dispatch 0x7f7e3c000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f7e2a7fc640 3 client.4304.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f7e2a7fc640 3 client.4304.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7f7e2a7fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 1 -- 172.21.15.53:0/2210446254 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c03b620 con 0x7fd8ac06d270 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: dump: 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.220 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 1 -- 172.21.15.53:0/2210446254 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c03f070 con 0x7fd8ac06d270 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 1 -- 172.21.15.53:0/2210446254 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c03ba10 con 0x7fd8ac06d270 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a17fa640 10 monclient: dump: 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.221 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 5 monclient: authenticate success, global_id 4362 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 10 monclient: _renew_subs 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac0a72d0 con 0x7fd8ac06d270 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 10 monclient: _renew_subs 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac1559d0 con 0x7fd8ac06d270 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 10 monclient: dump: 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 1 -- 172.21.15.53:0/1258558762 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc00a850 con 0x7f67e413f1e0 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 1 -- 172.21.15.53:0/1258558762 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc00ac10 con 0x7f67e413f1e0 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67cb7fe640 10 monclient: dump: 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.224 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.225 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.225 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8b386f640 1 librados: init done 2024-09-15T02:43:25.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.040+0000 7fd8a37fe640 4 set_mon_vals no callback set 2024-09-15T02:43:25.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7f67e17fa640 4 set_mon_vals no callback set 2024-09-15T02:43:25.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.041+0000 7ff6edffb640 1 -- 172.21.15.53:0/3582228471 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7ff6e8012860 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7fd8b386f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7fd8b386f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7fd8b386f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ac1556d0 con 0x7fd8ac06d270 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7ff6edffb640 1 --2- 172.21.15.53:0/3582228471 >> v2:172.21.15.104:6816/32804 conn(0x7ff6dc06e8c0 0x7ff6dc070d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7ff6edffb640 1 -- 172.21.15.53:0/3582228471 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7ff6e808bd70 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7ff6edffb640 10 client.4310.objecter ms_dispatch 0x7ff6f8000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7ff6edffb640 3 client.4310.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7ff6edffb640 3 client.4310.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7ff6edffb640 20 client.4310.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7ff6fd1da640 1 --2- 172.21.15.53:0/3582228471 >> v2:172.21.15.104:6816/32804 conn(0x7ff6dc06e8c0 0x7ff6dc070d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7f7a827fc640 1 -- 172.21.15.53:0/4114574554 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7a88077560 con 0x7f7a84008b70 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7f7a827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7f7a827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7ff6fd1da640 10 monclient: get_auth_request con 0x7ff6dc06e8c0 auth_method 0 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7f67e9f5d640 5 monclient: authenticate success, global_id 4313 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7f67e9f5d640 10 monclient: _renew_subs 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7f67e9f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.042+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e41398f0 con 0x7f67e413f1e0 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.043+0000 7f67e9f5d640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.043+0000 7f67e9f5d640 10 monclient: _renew_subs 2024-09-15T02:43:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.043+0000 7f67e9f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.043+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e40b0be0 con 0x7f67e413f1e0 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.043+0000 7f67e9f5d640 1 librados: init done 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7f67e9f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7f67e9f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7f67e9f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e40646e0 con 0x7f67e413f1e0 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.044+0000 7fd8a17fa640 1 -- 172.21.15.53:0/2210446254 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fd89c03a420 con 0x7fd8ac06d270 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.045+0000 7fd8a17fa640 1 --2- 172.21.15.53:0/2210446254 >> v2:172.21.15.104:6816/32804 conn(0x7fd89006e950 0x7fd890070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.045+0000 7fd8a17fa640 1 -- 172.21.15.53:0/2210446254 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7fd89c0ac4a0 con 0x7fd8ac06d270 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.045+0000 7fd8a17fa640 10 client.4362.objecter ms_dispatch 0x7fd8ac000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.045+0000 7fd8a17fa640 3 client.4362.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.045+0000 7fd8a17fa640 3 client.4362.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.045+0000 7fd8a17fa640 20 client.4362.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.046+0000 7fd8b160b640 1 --2- 172.21.15.53:0/2210446254 >> v2:172.21.15.104:6816/32804 conn(0x7fd89006e950 0x7fd890070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.046+0000 7fd8b160b640 10 monclient: get_auth_request con 0x7fd89006e950 auth_method 0 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.046+0000 7ff6fd1da640 1 --2- 172.21.15.53:0/3582228471 >> v2:172.21.15.104:6816/32804 conn(0x7ff6dc06e8c0 0x7ff6dc070d70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4034000 tx=0x7ff6f4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.046+0000 7ff6edffb640 10 client.4310.objecter ms_handle_connect 0x7ff6dc06e8c0 2024-09-15T02:43:25.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.046+0000 7fd8a17fa640 10 client.4362.objecter ms_handle_connect 0x7fd89006e950 2024-09-15T02:43:25.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.046+0000 7fd8b160b640 1 --2- 172.21.15.53:0/2210446254 >> v2:172.21.15.104:6816/32804 conn(0x7fd89006e950 0x7fd890070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009940 tx=0x7fd8a80091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.048+0000 7f7e2a7fc640 1 -- 172.21.15.53:0/1959147151 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7e300563b0 con 0x7f7e3c06d700 2024-09-15T02:43:25.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.048+0000 7f7e2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.048+0000 7f7e2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.050+0000 7ff6edffb640 1 -- 172.21.15.53:0/3582228471 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff6e8056300 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.050+0000 7ff6edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.050+0000 7ff6edffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.051+0000 7fd8a17fa640 1 -- 172.21.15.53:0/2210446254 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd89c076cf0 con 0x7fd8ac06d270 2024-09-15T02:43:25.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.051+0000 7fd8a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.051+0000 7fd8a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.056+0000 7f67cb7fe640 1 -- 172.21.15.53:0/1258558762 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f67dc024420 con 0x7f67e413f1e0 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.056+0000 7f67cb7fe640 1 --2- 172.21.15.53:0/1258558762 >> v2:172.21.15.104:6816/32804 conn(0x7f67d006e880 0x7f67d0070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.056+0000 7f67cb7fe640 1 -- 172.21.15.53:0/1258558762 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f67dc08c130 con 0x7f67e413f1e0 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.056+0000 7f67cb7fe640 10 client.4313.objecter ms_dispatch 0x7f67e4000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.056+0000 7f67cb7fe640 3 client.4313.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.056+0000 7f67cb7fe640 3 client.4313.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.056+0000 7f67cb7fe640 20 client.4313.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.060+0000 7f67e37fe640 1 --2- 172.21.15.53:0/1258558762 >> v2:172.21.15.104:6816/32804 conn(0x7f67d006e880 0x7f67d0070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.060+0000 7f67e37fe640 10 monclient: get_auth_request con 0x7f67d006e880 auth_method 0 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.060+0000 7f67cb7fe640 1 -- 172.21.15.53:0/1258558762 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f67dc056570 con 0x7f67e413f1e0 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.061+0000 7f67cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.061+0000 7f67cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.075+0000 7f67cb7fe640 10 client.4313.objecter ms_handle_connect 0x7f67d006e880 2024-09-15T02:43:25.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.075+0000 7f67e37fe640 1 --2- 172.21.15.53:0/1258558762 >> v2:172.21.15.104:6816/32804 conn(0x7f67d006e880 0x7f67d0070d30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67d800acc0 tx=0x7f67d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.423+0000 7fafdcc57640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T02:43:25.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.423+0000 7fafdcc57640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T02:43:25.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.423+0000 7fafdcc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.423+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fafd80646e0 con 0x7fafd813a560 2024-09-15T02:43:25.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.424+0000 7fa4b8109640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-15T02:43:25.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.424+0000 7fa4b8109640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-15T02:43:25.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.424+0000 7fa4b8109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.424+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa4b00a6a40 con 0x7fa4b00a98d0 2024-09-15T02:43:25.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.424+0000 7fafce7fc640 1 -- 172.21.15.53:0/141445028 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fafd0077750 con 0x7fafd813a560 2024-09-15T02:43:25.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.424+0000 7fafce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T02:43:25.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.424+0000 7fafce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.426 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:25.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.426+0000 7fa4a5ffb640 1 -- 172.21.15.53:0/2263763715 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa4a8059850 con 0x7fa4b00a98d0 2024-09-15T02:43:25.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.426+0000 7fa4a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-15T02:43:25.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.426+0000 7fa4a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.426+0000 7f7a931ed640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T02:43:25.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.426+0000 7f7a931ed640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T02:43:25.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.426+0000 7f7a931ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.426+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7a74005590 con 0x7f7a84008b70 2024-09-15T02:43:25.428 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:25.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.427+0000 7fafdcc57640 10 librados: watch_flush enter 2024-09-15T02:43:25.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.427+0000 7fafdcc57640 10 librados: watch_flush exit 2024-09-15T02:43:25.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.427+0000 7f7a827fc640 1 -- 172.21.15.53:0/4114574554 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a8807ad70 con 0x7f7a84008b70 2024-09-15T02:43:25.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.427+0000 7f7a827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T02:43:25.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.427+0000 7f7a827fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.427+0000 7fafdcc57640 20 client.4350.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.427+0000 7fafdcc57640 10 client.4350.objecter successfully canceled tick 2024-09-15T02:43:25.430 INFO:teuthology.orchestra.run.smithi053.stdout:55834574851 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 >> v2:172.21.15.104:6816/32804 conn(0x7fafc006e8d0 msgr2=0x7fafc0070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7fafdcc57640 1 --2- 172.21.15.53:0/141445028 >> v2:172.21.15.104:6816/32804 conn(0x7fafc006e8d0 0x7fafc0070d80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fafc80016a0 tx=0x7fafc8007ab0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7fafdcc57640 10 monclient: shutdown 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7fafdcc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafd813a560 msgr2=0x7fafd8154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7fafdcc57640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafd813a560 0x7fafd8154f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafd000ab00 tx=0x7fafd0039000 comp rx=0 tx=0).stop 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7f6a6399d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7f6a6399d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T02:43:25.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7f6a6399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.428+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6a5c06e480 con 0x7f6a5c0b0870 2024-09-15T02:43:25.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 shutdown_connections 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafd813a560 0x7fafd8154f90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 --2- 172.21.15.53:0/141445028 >> v2:172.21.15.104:6816/32804 conn(0x7fafc006e8d0 0x7fafc0070d80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafd81398b0 0x7fafd80a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 --2- 172.21.15.53:0/141445028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafd806a8c0 0x7fafd80a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 >> 172.21.15.53:0/141445028 conn(0x7fafd8067b60 msgr2=0x7fafd80b0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 shutdown_connections 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 -- 172.21.15.53:0/141445028 wait complete. 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fa4b8109640 10 librados: watch_flush enter 2024-09-15T02:43:25.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fa4b8109640 10 librados: watch_flush exit 2024-09-15T02:43:25.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fafdcc57640 1 librados: shutdown 2024-09-15T02:43:25.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.429+0000 7fa4b8109640 20 client.4301.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7fa4b8109640 10 client.4301.objecter successfully canceled tick 2024-09-15T02:43:25.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f6a517fa640 1 -- 172.21.15.53:0/989860089 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a5c06e480 con 0x7f6a5c0b0870 2024-09-15T02:43:25.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f6a517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T02:43:25.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f6a517fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.435 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:25.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 >> v2:172.21.15.104:6816/32804 conn(0x7fa49406e8d0 msgr2=0x7fa494070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7fa4b8109640 1 --2- 172.21.15.53:0/2263763715 >> v2:172.21.15.104:6816/32804 conn(0x7fa49406e8d0 0x7fa494070d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4b006ec40 tx=0x7fa4a0008040 comp rx=0 tx=0).stop 2024-09-15T02:43:25.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7fa4b8109640 10 monclient: shutdown 2024-09-15T02:43:25.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7fa4b8109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f7a931ed640 10 librados: watch_flush enter 2024-09-15T02:43:25.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f7a931ed640 10 librados: watch_flush exit 2024-09-15T02:43:25.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f7a931ed640 20 client.4353.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f7a931ed640 10 client.4353.objecter successfully canceled tick 2024-09-15T02:43:25.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa4b00a98d0 msgr2=0x7fa4b00a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7fa4b8109640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa4b00a98d0 0x7fa4b00a6230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800cec0 tx=0x7fa4a8001c50 comp rx=0 tx=0).stop 2024-09-15T02:43:25.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 >> v2:172.21.15.104:6816/32804 conn(0x7f7a7006e8d0 msgr2=0x7f7a70070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f7a931ed640 1 --2- 172.21.15.53:0/4114574554 >> v2:172.21.15.104:6816/32804 conn(0x7f7a7006e8d0 0x7f7a70070d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c066cc0 tx=0x7f7a8c064d50 comp rx=0 tx=0).stop 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.430+0000 7f7a931ed640 10 monclient: shutdown 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a84008b70 msgr2=0x7f7a840be940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 --2- 172.21.15.53:0/4114574554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a84008b70 0x7f7a840be940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009210 tx=0x7f7a88032000 comp rx=0 tx=0).stop 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 shutdown_connections 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 --2- 172.21.15.53:0/2263763715 >> v2:172.21.15.104:6816/32804 conn(0x7fa49406e8d0 0x7fa494070d80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4b0141300 0x7fa4b00a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4b00a9ca0 0x7fa4b00a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 --2- 172.21.15.53:0/2263763715 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa4b00a98d0 0x7fa4b00a6230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 >> 172.21.15.53:0/2263763715 conn(0x7fa4b0067b60 msgr2=0x7fa4b00aaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 shutdown_connections 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 --2- 172.21.15.53:0/4114574554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a84008b70 0x7f7a840be940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 --2- 172.21.15.53:0/4114574554 >> v2:172.21.15.104:6816/32804 conn(0x7f7a7006e8d0 0x7f7a70070d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 --2- 172.21.15.53:0/4114574554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a84008740 0x7f7a8402d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 --2- 172.21.15.53:0/4114574554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a840073b0 0x7f7a8402ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 >> 172.21.15.53:0/4114574554 conn(0x7f7a84037ce0 msgr2=0x7f7a84006940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 shutdown_connections 2024-09-15T02:43:25.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 -- 172.21.15.53:0/2263763715 wait complete. 2024-09-15T02:43:25.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 shutdown_connections 2024-09-15T02:43:25.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7fa4b8109640 1 librados: shutdown 2024-09-15T02:43:25.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 -- 172.21.15.53:0/4114574554 wait complete. 2024-09-15T02:43:25.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.431+0000 7f7a931ed640 1 librados: shutdown 2024-09-15T02:43:25.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.432+0000 7ff6fe9dd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-15T02:43:25.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.432+0000 7ff6fe9dd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-15T02:43:25.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.432+0000 7ff6fe9dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.432+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff6f002ef60 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f7e40a4f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T02:43:25.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f7e40a4f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T02:43:25.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f7e40a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7e3c133960 con 0x7f7e3c06d700 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f6a6399d640 10 librados: watch_flush enter 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f6a6399d640 10 librados: watch_flush exit 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f6a6399d640 20 client.4307.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.433+0000 7f6a6399d640 10 client.4307.objecter successfully canceled tick 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 >> v2:172.21.15.104:6816/32804 conn(0x7f6a4006e8d0 msgr2=0x7f6a40070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 --2- 172.21.15.53:0/989860089 >> v2:172.21.15.104:6816/32804 conn(0x7f6a4006e8d0 0x7f6a40070d80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009740 tx=0x7f6a58006c50 comp rx=0 tx=0).stop 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 10 monclient: shutdown 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a5c0b0870 msgr2=0x7f6a5c150990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a5c0b0870 0x7f6a5c150990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00b4c0 tx=0x7f6a4c006f50 comp rx=0 tx=0).stop 2024-09-15T02:43:25.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 shutdown_connections 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 --2- 172.21.15.53:0/989860089 >> v2:172.21.15.104:6816/32804 conn(0x7f6a4006e8d0 0x7f6a40070d80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a5c0b0c40 0x7f6a5c06dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a5c0b0870 0x7f6a5c150990 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 --2- 172.21.15.53:0/989860089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a5c0abcb0 0x7f6a5c13fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 >> 172.21.15.53:0/989860089 conn(0x7f6a5c067b00 msgr2=0x7f6a5c0af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 shutdown_connections 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f6a6399d640 1 -- 172.21.15.53:0/989860089 wait complete. 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f7e2a7fc640 1 -- 172.21.15.53:0/1959147151 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e30059bc0 con 0x7f7e3c06d700 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f7e2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T02:43:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7f7e2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.435+0000 7f6a6399d640 1 librados: shutdown 2024-09-15T02:43:25.446 INFO:teuthology.orchestra.run.smithi053.stdout:55834574851 2024-09-15T02:43:25.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7ff6edffb640 1 -- 172.21.15.53:0/3582228471 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff6e8059b10 con 0x7ff6f00bb4a0 2024-09-15T02:43:25.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7ff6edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-15T02:43:25.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.434+0000 7ff6edffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.447 INFO:teuthology.orchestra.run.smithi053.stdout:55834574851 2024-09-15T02:43:25.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.436+0000 7fd8b386f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T02:43:25.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.436+0000 7fd8b386f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T02:43:25.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.436+0000 7fd8b386f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:25.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.436+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd8ac0646e0 con 0x7fd8ac06d270 2024-09-15T02:43:25.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.436+0000 7fd8a17fa640 1 -- 172.21.15.53:0/2210446254 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd89c076cf0 con 0x7fd8ac06d270 2024-09-15T02:43:25.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.436+0000 7fd8a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T02:43:25.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.436+0000 7fd8a17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.448 INFO:teuthology.orchestra.run.smithi053.stdout:55834574853 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.438+0000 7f7e40a4f640 10 librados: watch_flush enter 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.438+0000 7f7e40a4f640 10 librados: watch_flush exit 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.438+0000 7f7e40a4f640 20 client.4304.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.438+0000 7f7e40a4f640 10 client.4304.objecter successfully canceled tick 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.438+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 >> v2:172.21.15.104:6816/32804 conn(0x7f7e1c06e8d0 msgr2=0x7f7e1c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/1959147151 >> v2:172.21.15.104:6816/32804 conn(0x7f7e1c06e8d0 0x7f7e1c070d80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c039040 tx=0x7f7e2c036000 comp rx=0 tx=0).stop 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 10 monclient: shutdown 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06d700 msgr2=0x7f7e3c14cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06d700 0x7f7e3c14cb50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c810 tx=0x7f7e30007e10 comp rx=0 tx=0).stop 2024-09-15T02:43:25.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 shutdown_connections 2024-09-15T02:43:25.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e3c06dad0 0x7f7e3c133cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e3c06d700 0x7f7e3c14cb50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/1959147151 >> v2:172.21.15.104:6816/32804 conn(0x7f7e1c06e8d0 0x7f7e1c070d80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 --2- 172.21.15.53:0/1959147151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e3c06d090 0x7f7e3c14c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 >> 172.21.15.53:0/1959147151 conn(0x7f7e3c063810 msgr2=0x7f7e3c137d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7ff6fe9dd640 10 librados: watch_flush enter 2024-09-15T02:43:25.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 shutdown_connections 2024-09-15T02:43:25.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7ff6fe9dd640 10 librados: watch_flush exit 2024-09-15T02:43:25.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 -- 172.21.15.53:0/1959147151 wait complete. 2024-09-15T02:43:25.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7ff6fe9dd640 20 client.4310.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7ff6fe9dd640 10 client.4310.objecter successfully canceled tick 2024-09-15T02:43:25.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7f7e40a4f640 1 librados: shutdown 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.439+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 >> v2:172.21.15.104:6816/32804 conn(0x7ff6dc06e8c0 msgr2=0x7ff6dc070d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/3582228471 >> v2:172.21.15.104:6816/32804 conn(0x7ff6dc06e8c0 0x7ff6dc070d70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4034000 tx=0x7ff6f4032000 comp rx=0 tx=0).stop 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 10 monclient: shutdown 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6f00bb4a0 msgr2=0x7ff6f002ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6f00bb4a0 0x7ff6f002ca20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0003d90 tx=0x7ff6e8009040 comp rx=0 tx=0).stop 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 shutdown_connections 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6f00e6680 0x7ff6f00d6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/3582228471 >> v2:172.21.15.104:6816/32804 conn(0x7ff6dc06e8c0 0x7ff6dc070d70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6f002d4f0 0x7ff6f00e6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 --2- 172.21.15.53:0/3582228471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6f00bb4a0 0x7ff6f002ca20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.440+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 >> 172.21.15.53:0/3582228471 conn(0x7ff6f0039cd0 msgr2=0x7ff6f00300e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 10 librados: watch_flush enter 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 10 librados: watch_flush exit 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 20 client.4362.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 10 client.4362.objecter successfully canceled tick 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 >> v2:172.21.15.104:6816/32804 conn(0x7fd89006e950 msgr2=0x7fd890070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 1 --2- 172.21.15.53:0/2210446254 >> v2:172.21.15.104:6816/32804 conn(0x7fd89006e950 0x7fd890070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009940 tx=0x7fd8a80091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 10 monclient: shutdown 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8ac06d270 msgr2=0x7fd8ac0a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 shutdown_connections 2024-09-15T02:43:25.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7ff6fe9dd640 1 -- 172.21.15.53:0/3582228471 wait complete. 2024-09-15T02:43:25.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7ff6fe9dd640 1 librados: shutdown 2024-09-15T02:43:25.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.441+0000 7fd8b386f640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8ac06d270 0x7fd8ac0a54c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd89c031000 tx=0x7fd89c035000 comp rx=0 tx=0).stop 2024-09-15T02:43:25.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 shutdown_connections 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 --2- 172.21.15.53:0/2210446254 >> v2:172.21.15.104:6816/32804 conn(0x7fd89006e950 0x7fd890070e00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ac0a5a00 0x7fd8ac0a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8ac06d270 0x7fd8ac0a54c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 --2- 172.21.15.53:0/2210446254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd8ac154e60 0x7fd8ac06cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 >> 172.21.15.53:0/2210446254 conn(0x7fd8ac067b00 msgr2=0x7fd8ac0ab2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 shutdown_connections 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 -- 172.21.15.53:0/2210446254 wait complete. 2024-09-15T02:43:25.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.442+0000 7fd8b386f640 1 librados: shutdown 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.449+0000 7f67e9f5d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.449+0000 7f67e9f5d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.449+0000 7f67e9f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.449+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f67e4133e40 con 0x7f67e413f1e0 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.450+0000 7f67cb7fe640 1 -- 172.21.15.53:0/1258558762 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f67dc059f50 con 0x7f67e413f1e0 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.450+0000 7f67cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.450+0000 7f67cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:25.459 INFO:teuthology.orchestra.run.smithi053.stdout:55834574852 2024-09-15T02:43:25.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.453+0000 7f67e9f5d640 10 librados: watch_flush enter 2024-09-15T02:43:25.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 10 librados: watch_flush exit 2024-09-15T02:43:25.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 20 client.4313.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 10 client.4313.objecter successfully canceled tick 2024-09-15T02:43:25.461 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2024-09-15T02:43:25.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 >> v2:172.21.15.104:6816/32804 conn(0x7f67d006e880 msgr2=0x7f67d0070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/1258558762 >> v2:172.21.15.104:6816/32804 conn(0x7f67d006e880 0x7f67d0070d30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67d800acc0 tx=0x7f67d8008040 comp rx=0 tx=0).stop 2024-09-15T02:43:25.462 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 10 monclient: shutdown 2024-09-15T02:43:25.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 msgr2=0x7f67e4134640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e4134640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00c840 tx=0x7f67dc0015f0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2024-09-15T02:43:25.463 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 shutdown_connections 2024-09-15T02:43:25.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f67e413f1e0 0x7f67e4134640 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/1258558762 >> v2:172.21.15.104:6816/32804 conn(0x7f67d006e880 0x7f67d0070d30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67e413ee10 0x7f67e4134100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 --2- 172.21.15.53:0/1258558762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67e40ad1a0 0x7f67e40b1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.454+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 >> 172.21.15.53:0/1258558762 conn(0x7f67e4063810 msgr2=0x7f67e413d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2024-09-15T02:43:25.464 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.455+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 shutdown_connections 2024-09-15T02:43:25.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.455+0000 7f67e9f5d640 1 -- 172.21.15.53:0/1258558762 wait complete. 2024-09-15T02:43:25.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.455+0000 7f67e9f5d640 1 librados: shutdown 2024-09-15T02:43:25.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-09-15T02:43:25.466 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2024-09-15T02:43:25.467 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2024-09-15T02:43:25.467 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.468 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.6 2024-09-15T02:43:25.468 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2024-09-15T02:43:25.470 DEBUG:teuthology.parallel:result is None 2024-09-15T02:43:25.470 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T02:43:25.470 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:43:25.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.605+0000 7efe768c0640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.605+0000 7efe768c0640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.605+0000 7efe768c0640 10 monclient: monmap: 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.606535+0000 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.606535+0000 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.607 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.608 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.606+0000 7efe768c0640 1 Processor -- start 2024-09-15T02:43:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.606+0000 7efe768c0640 1 -- start start 2024-09-15T02:43:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.606+0000 7efe768c0640 10 monclient: init 2024-09-15T02:43:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T02:43:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe700b2c40 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe70068c50 0x7efe70068010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient(hunting): picked mon.noname-b con 0x7efe70068c50 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efe7006df40 0x7efe7006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient(hunting): picked mon.noname-c con 0x7efe7006df40 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7006ec40 con 0x7efe700b2c40 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe70068550 con 0x7efe7006df40 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe70069cb0 con 0x7efe70068c50 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe768c0640 10 monclient(hunting): authenticate will time out at 742.177201s 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe70068c50 0x7efe70068010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe70068c50 0x7efe70068010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47236/0 (socket says 172.21.15.53:47236) 2024-09-15T02:43:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 -- 172.21.15.53:0/2788237614 learned_addr learned my addr 172.21.15.53:0/2788237614 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): get_auth_request con 0x7efe70068c50 auth_method 0 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe74e5d640 1 --2- 172.21.15.53:0/2788237614 >> v2:172.21.15.104:3300/0 conn(0x7efe7006df40 0x7efe7006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe74e5d640 10 monclient(hunting): get_auth_request con 0x7efe7006df40 auth_method 0 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe74e5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe74e5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe74e5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 -- 172.21.15.53:0/2788237614 >> v2:172.21.15.104:3300/0 conn(0x7efe7006df40 msgr2=0x7efe7006e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 --2- 172.21.15.53:0/2788237614 >> v2:172.21.15.104:3300/0 conn(0x7efe7006df40 0x7efe7006e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 -- 172.21.15.53:0/2788237614 >> v1:172.21.15.104:6789/0 conn(0x7efe700b2c40 legacy=0x7efe70140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 monclient: found mon.noname-b 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.607+0000 7efe6f7fe640 1 -- 172.21.15.53:0/2788237614 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70069eb0 con 0x7efe70068c50 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6f7fe640 1 --2- 172.21.15.53:0/2788237614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe70068c50 0x7efe70068010 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efe64009910 tx=0x7efe640091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aed37c50c1c1e149 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 1 -- 172.21.15.53:0/2788237614 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6402ad20 con 0x7efe70068c50 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: dump: 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 1 -- 172.21.15.53:0/2788237614 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6402ae80 con 0x7efe70068c50 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe768c0640 5 monclient: authenticate success, global_id 4319 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe768c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe768c0640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 1 -- 172.21.15.53:0/2788237614 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64032670 con 0x7efe70068c50 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe768c0640 4 set_mon_vals no callback set 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.608+0000 7efe6effd640 10 monclient: dump: 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 10 monclient: shutdown 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 -- 172.21.15.53:0/2788237614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe70068c50 msgr2=0x7efe70068010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 --2- 172.21.15.53:0/2788237614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe70068c50 0x7efe70068010 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efe64009910 tx=0x7efe640091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 -- 172.21.15.53:0/2788237614 shutdown_connections 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 --2- 172.21.15.53:0/2788237614 >> v2:172.21.15.104:3300/0 conn(0x7efe7006df40 0x7efe7006e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 --2- 172.21.15.53:0/2788237614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe70068c50 0x7efe70068010 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 -- 172.21.15.53:0/2788237614 >> 172.21.15.53:0/2788237614 conn(0x7efe70063810 msgr2=0x7efe700b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 -- 172.21.15.53:0/2788237614 shutdown_connections 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 -- 172.21.15.53:0/2788237614 wait complete. 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 10 monclient: monmap: 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.626 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 librados: starting msgr at 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.609+0000 7efe768c0640 1 librados: starting objecter 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 Processor -- start 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 -- start start 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 librados: setting wanted keys 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 librados: calling monclient init 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient: init 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe70068c50 0x7efe70151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): picked mon.noname-a con 0x7efe70068c50 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe7006df40 0x7efe701515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): picked mon.noname-b con 0x7efe7006df40 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe700b2c40 0x7efe7013bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): picked mon.noname-c con 0x7efe700b2c40 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe700abbe0 con 0x7efe70068c50 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7013b670 con 0x7efe7006df40 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7013b830 con 0x7efe700b2c40 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe768c0640 10 monclient(hunting): authenticate will time out at 742.180829s 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe74e5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe700b2c40 0x7efe7013bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe6ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe70068c50 0x7efe70151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe74e5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe700b2c40 0x7efe7013bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47238/0 (socket says 172.21.15.53:47238) 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe74e5d640 1 -- 172.21.15.53:0/2426887563 learned_addr learned my addr 172.21.15.53:0/2426887563 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe74e5d640 10 monclient(hunting): get_auth_request con 0x7efe700b2c40 auth_method 0 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe74e5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe74e5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe74e5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe6ffff640 10 monclient(hunting): get_auth_request con 0x7efe70068c50 auth_method 0 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.610+0000 7efe6f7fe640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe7006df40 0x7efe701515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6f7fe640 10 monclient(hunting): get_auth_request con 0x7efe7006df40 auth_method 0 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 1 -- 172.21.15.53:0/2426887563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe7006df40 msgr2=0x7efe701515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe7006df40 0x7efe701515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 1 -- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe70068c50 msgr2=0x7efe70151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe70068c50 0x7efe70151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 1 -- 172.21.15.53:0/2426887563 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe64009450 con 0x7efe700b2c40 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6ffff640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe70068c50 0x7efe70151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe74e5d640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe700b2c40 0x7efe7013bd40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efe68009fa0 tx=0x7efe68009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 client.?.objecter ms_handle_connect 0x7efe700b2c40 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 1 -- 172.21.15.53:0/2426887563 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6800ee60 con 0x7efe700b2c40 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: dump: 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 5 monclient: authenticate success, global_id 4322 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 1 -- 172.21.15.53:0/2426887563 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe68014420 con 0x7efe700b2c40 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 10 monclient: _renew_subs 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7013c870 con 0x7efe700b2c40 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 1 -- 172.21.15.53:0/2426887563 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe680147e0 con 0x7efe700b2c40 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe57fff640 10 monclient: dump: 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.633 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 10 monclient: _renew_subs 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe700ac9d0 con 0x7efe700b2c40 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe768c0640 1 librados: init done 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.611+0000 7efe6e7fc640 4 set_mon_vals no callback set 2024-09-15T02:43:25.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.612+0000 7efe768c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.612+0000 7efe768c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.612+0000 7efe768c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.612+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe700646e0 con 0x7efe700b2c40 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe57fff640 1 -- 172.21.15.53:0/2426887563 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7efe68014b50 con 0x7efe700b2c40 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe57fff640 1 --2- 172.21.15.53:0/2426887563 >> v2:172.21.15.104:6816/32804 conn(0x7efe5c06e800 0x7efe5c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe57fff640 1 -- 172.21.15.53:0/2426887563 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7efe6808bd50 con 0x7efe700b2c40 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe57fff640 10 client.4322.objecter ms_dispatch 0x7efe70000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe57fff640 3 client.4322.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe57fff640 3 client.4322.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe57fff640 20 client.4322.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.613+0000 7efe6ffff640 1 --2- 172.21.15.53:0/2426887563 >> v2:172.21.15.104:6816/32804 conn(0x7efe5c06e800 0x7efe5c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.614+0000 7efe6ffff640 10 monclient: get_auth_request con 0x7efe5c06e800 auth_method 0 2024-09-15T02:43:25.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.614+0000 7efe57fff640 10 client.4322.objecter ms_handle_connect 0x7efe5c06e800 2024-09-15T02:43:25.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.614+0000 7efe6ffff640 1 --2- 172.21.15.53:0/2426887563 >> v2:172.21.15.104:6816/32804 conn(0x7efe5c06e800 0x7efe5c070cb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe60007fd0 tx=0x7efe60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.617+0000 7efe57fff640 1 -- 172.21.15.53:0/2426887563 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efe6808f030 con 0x7efe700b2c40 2024-09-15T02:43:25.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.617+0000 7efe57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.617+0000 7efe57fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:25.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.776+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe700b0ce0 con 0x7efe5c06e800 2024-09-15T02:43:25.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.779+0000 7efe57fff640 1 -- 172.21.15.53:0/2426887563 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48641 (secure 0 0 0) 0x7efe700b0ce0 con 0x7efe5c06e800 2024-09-15T02:43:25.781 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:43:25.781 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:43:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 10 librados: watch_flush enter 2024-09-15T02:43:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 10 librados: watch_flush exit 2024-09-15T02:43:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 20 client.4322.objecter shutdown clearing up homeless session... 2024-09-15T02:43:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 10 client.4322.objecter successfully canceled tick 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 >> v2:172.21.15.104:6816/32804 conn(0x7efe5c06e800 msgr2=0x7efe5c070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 --2- 172.21.15.53:0/2426887563 >> v2:172.21.15.104:6816/32804 conn(0x7efe5c06e800 0x7efe5c070cb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe60007fd0 tx=0x7efe60009040 comp rx=0 tx=0).stop 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 10 monclient: shutdown 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe700b2c40 msgr2=0x7efe7013bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe700b2c40 0x7efe7013bd40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efe68009fa0 tx=0x7efe68009900 comp rx=0 tx=0).stop 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 shutdown_connections 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe700b2c40 0x7efe7013bd40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 --2- 172.21.15.53:0/2426887563 >> v2:172.21.15.104:6816/32804 conn(0x7efe5c06e800 0x7efe5c070cb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe7006df40 0x7efe701515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 --2- 172.21.15.53:0/2426887563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe70068c50 0x7efe70151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 >> 172.21.15.53:0/2426887563 conn(0x7efe70063810 msgr2=0x7efe700ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 shutdown_connections 2024-09-15T02:43:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 -- 172.21.15.53:0/2426887563 wait complete. 2024-09-15T02:43:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.781+0000 7efe768c0640 1 librados: shutdown 2024-09-15T02:43:25.793 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-15T02:43:25.330543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2380192,"kb_used_data":1312,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752594528,"statfs":{"total":773094113280,"available":770656796672,"internally_reserved":0,"allocated":1343488,"data_stored":1444582,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.626528+0000","last_change":"2024-09-15T02:43:23.626724+0000","last_active":"2024-09-15T02:43:23.626528+0000","last_peered":"2024-09-15T02:43:23.626528+0000","last_clean":"2024-09-15T02:43:23.626528+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:43:23.626528+0000","last_undegraded":"2024-09-15T02:43:23.626528+0000","last_fullsized":"2024-09-15T02:43:23.626528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:48.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.425756+0000","last_change":"2024-09-15T02:43:23.426048+0000","last_active":"2024-09-15T02:43:23.425756+0000","last_peered":"2024-09-15T02:43:23.425756+0000","last_clean":"2024-09-15T02:43:23.425756+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:43:23.425756+0000","last_undegraded":"2024-09-15T02:43:23.425756+0000","last_fullsized":"2024-09-15T02:43:23.425756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:19.460873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.420262+0000","last_change":"2024-09-15T02:43:23.420483+0000","last_active":"2024-09-15T02:43:23.420262+0000","last_peered":"2024-09-15T02:43:23.420262+0000","last_clean":"2024-09-15T02:43:23.420262+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:43:23.420262+0000","last_undegraded":"2024-09-15T02:43:23.420262+0000","last_fullsized":"2024-09-15T02:43:23.420262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:03:14.233618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.425804+0000","last_change":"2024-09-15T02:43:23.426071+0000","last_active":"2024-09-15T02:43:23.425804+0000","last_peered":"2024-09-15T02:43:23.425804+0000","last_clean":"2024-09-15T02:43:23.425804+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:43:23.425804+0000","last_undegraded":"2024-09-15T02:43:23.425804+0000","last_fullsized":"2024-09-15T02:43:23.425804+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:00:03.954753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.423983+0000","last_change":"2024-09-15T02:43:23.423983+0000","last_active":"2024-09-15T02:43:23.423983+0000","last_peered":"2024-09-15T02:43:23.423983+0000","last_clean":"2024-09-15T02:43:23.423983+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:43:23.423983+0000","last_undegraded":"2024-09-15T02:43:23.423983+0000","last_fullsized":"2024-09-15T02:43:23.423983+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:45.683030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.419705+0000","last_change":"2024-09-15T02:43:23.419938+0000","last_active":"2024-09-15T02:43:23.419705+0000","last_peered":"2024-09-15T02:43:23.419705+0000","last_clean":"2024-09-15T02:43:23.419705+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:43:23.419705+0000","last_undegraded":"2024-09-15T02:43:23.419705+0000","last_fullsized":"2024-09-15T02:43:23.419705+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:09:35.753407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.420211+0000","last_change":"2024-09-15T02:43:23.420430+0000","last_active":"2024-09-15T02:43:23.420211+0000","last_peered":"2024-09-15T02:43:23.420211+0000","last_clean":"2024-09-15T02:43:23.420211+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:43:23.420211+0000","last_undegraded":"2024-09-15T02:43:23.420211+0000","last_fullsized":"2024-09-15T02:43:23.420211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:05:17.337850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.423233+0000","last_change":"2024-09-15T02:43:23.423477+0000","last_active":"2024-09-15T02:43:23.423233+0000","last_peered":"2024-09-15T02:43:23.423233+0000","last_clean":"2024-09-15T02:43:23.423233+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:43:23.423233+0000","last_undegraded":"2024-09-15T02:43:23.423233+0000","last_fullsized":"2024-09-15T02:43:23.423233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:05:30.605263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":101,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-15T02:43:22.410993+0000","last_change":"2024-09-15T02:43:20.387731+0000","last_active":"2024-09-15T02:43:22.410993+0000","last_peered":"2024-09-15T02:43:22.410993+0000","last_clean":"2024-09-15T02:43:22.410993+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:43:22.410993+0000","last_undegraded":"2024-09-15T02:43:22.410993+0000","last_fullsized":"2024-09-15T02:43:22.410993+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_clean_scrub_stamp":"2024-09-15T02:43:19.374225+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:33:29.816927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":212992,"data_stored":579614,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":147456,"data_stored":45986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":163840,"data_stored":50705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297500,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":147456,"data_stored":45986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":147456,"data_stored":45986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":50686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":163840,"data_stored":50705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":200704,"data_stored":574914,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:43:25.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:43:25.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.889+0000 7f9e62494640 10 monclient: get_monmap_and_config 2024-09-15T02:43:25.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.889+0000 7f9e62494640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.889+0000 7f9e62494640 10 monclient: monmap: 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:25.890245+0000 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:25.890245+0000 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:25.891 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 1 Processor -- start 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 1 -- start start 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient: init 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e5c0ae210 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e5c0ae5e0 0x7f9e5c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient(hunting): picked mon.noname-c con 0x7f9e5c0ae5e0 addr v2:172.21.15.104:3300/0 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c06f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient(hunting): picked mon.noname-b con 0x7f9e5c0aca00 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c13b4f0 con 0x7f9e5c0ae210 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c0aa770 con 0x7f9e5c0ae5e0 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0acf80 con 0x7f9e5c0aca00 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.890+0000 7f9e62494640 10 monclient(hunting): authenticate will time out at 742.460919s 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c06f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c06f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47242/0 (socket says 172.21.15.53:47242) 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 -- 172.21.15.53:0/3019836106 learned_addr learned my addr 172.21.15.53:0/3019836106 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): get_auth_request con 0x7f9e5c0aca00 auth_method 0 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e5b7fe640 1 --2- 172.21.15.53:0/3019836106 >> v2:172.21.15.104:3300/0 conn(0x7f9e5c0ae5e0 0x7f9e5c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e5b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e5c0ae5e0 auth_method 0 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 -- 172.21.15.53:0/3019836106 >> v2:172.21.15.104:3300/0 conn(0x7f9e5c0ae5e0 msgr2=0x7f9e5c06ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 --2- 172.21.15.53:0/3019836106 >> v2:172.21.15.104:3300/0 conn(0x7f9e5c0ae5e0 0x7f9e5c06ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 -- 172.21.15.53:0/3019836106 >> v1:172.21.15.104:6789/0 conn(0x7f9e5c0ae210 legacy=0x7f9e5c140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 monclient: found mon.noname-b 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.891+0000 7f9e60a31640 1 -- 172.21.15.53:0/3019836106 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0aa970 con 0x7f9e5c0aca00 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e60a31640 1 --2- 172.21.15.53:0/3019836106 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c06f380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009f20 tx=0x7f9e54009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b318a1eb6269ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 1 -- 172.21.15.53:0/3019836106 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54033410 con 0x7f9e5c0aca00 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: dump: 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.902 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 1 -- 172.21.15.53:0/3019836106 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54033570 con 0x7f9e5c0aca00 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 5 monclient: authenticate success, global_id 4325 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 1 -- 172.21.15.53:0/3019836106 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54033950 con 0x7f9e5c0aca00 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e5affd640 10 monclient: dump: 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 4 set_mon_vals no callback set 2024-09-15T02:43:25.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 10 monclient: shutdown 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 -- 172.21.15.53:0/3019836106 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 msgr2=0x7f9e5c06f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 --2- 172.21.15.53:0/3019836106 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c06f380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009f20 tx=0x7f9e54009840 comp rx=0 tx=0).stop 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 -- 172.21.15.53:0/3019836106 shutdown_connections 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 --2- 172.21.15.53:0/3019836106 >> v2:172.21.15.104:3300/0 conn(0x7f9e5c0ae5e0 0x7f9e5c06ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 --2- 172.21.15.53:0/3019836106 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c06f380 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 -- 172.21.15.53:0/3019836106 >> 172.21.15.53:0/3019836106 conn(0x7f9e5c068590 msgr2=0x7f9e5c068990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 -- 172.21.15.53:0/3019836106 shutdown_connections 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.892+0000 7f9e62494640 1 -- 172.21.15.53:0/3019836106 wait complete. 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.893+0000 7f9e62494640 10 monclient: build_initial_monmap 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.893+0000 7f9e62494640 10 monclient: monmap: 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:25.905 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.893+0000 7f9e62494640 1 librados: starting msgr at 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.893+0000 7f9e62494640 1 librados: starting objecter 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.893+0000 7f9e62494640 1 Processor -- start 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 -- start start 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 librados: setting wanted keys 2024-09-15T02:43:25.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 librados: calling monclient init 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient: init 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c0b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): picked mon.noname-c con 0x7f9e5c0aca00 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e5c0ae210 0x7f9e5c150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): picked mon.noname-b con 0x7f9e5c0ae210 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e5c0ae5e0 0x7f9e5c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): picked mon.noname-a con 0x7f9e5c0ae5e0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0b1dd0 con 0x7f9e5c0ae5e0 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c119ce0 con 0x7f9e5c0ae210 2024-09-15T02:43:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c119f50 con 0x7f9e5c0aca00 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e62494640 10 monclient(hunting): authenticate will time out at 742.464691s 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c0b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e5c0ae5e0 0x7f9e5c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c0b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:47252/0 (socket says 172.21.15.53:47252) 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 1 -- 172.21.15.53:0/3256110666 learned_addr learned my addr 172.21.15.53:0/3256110666 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 10 monclient(hunting): get_auth_request con 0x7f9e5c0aca00 auth_method 0 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 10 monclient(hunting): get_auth_request con 0x7f9e5c0ae5e0 auth_method 0 2024-09-15T02:43:25.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5b7fe640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e5c0ae210 0x7f9e5c150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e5c0ae210 auth_method 0 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e60a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.894+0000 7f9e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 1 -- 172.21.15.53:0/3256110666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e5c0ae210 msgr2=0x7f9e5c150f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e5c0ae210 0x7f9e5c150f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 1 -- 172.21.15.53:0/3256110666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e5c0ae5e0 msgr2=0x7f9e5c151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e5c0ae5e0 0x7f9e5c151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 1 monclient: found mon.noname-c 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 1 -- 172.21.15.53:0/3256110666 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54009ae0 con 0x7f9e5c0aca00 2024-09-15T02:43:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5bfff640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c0b1000 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c610 tx=0x7f9e4c007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 client.?.objecter ms_handle_connect 0x7f9e5c0aca00 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 1 -- 172.21.15.53:0/3256110666 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c012630 con 0x7f9e5c0aca00 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: dump: 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: _finish_auth 0 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 5 monclient: authenticate success, global_id 4328 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 1 -- 172.21.15.53:0/3256110666 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c0127b0 con 0x7f9e5c0aca00 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 10 monclient: _renew_subs 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 1 -- 172.21.15.53:0/3256110666 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c012b70 con 0x7f9e5c0aca00 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c118d60 con 0x7f9e5c0aca00 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e43fff640 10 monclient: dump: 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e5a7fc640 4 set_mon_vals no callback set 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 10 monclient: _renew_subs 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c151bc0 con 0x7f9e5c0aca00 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.895+0000 7f9e62494640 1 librados: init done 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.896+0000 7f9e62494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.896+0000 7f9e62494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.896+0000 7f9e62494640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.896+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c11a110 con 0x7f9e5c0aca00 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.896+0000 7f9e43fff640 1 -- 172.21.15.53:0/3256110666 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f9e4c01a3b0 con 0x7f9e5c0aca00 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e43fff640 1 --2- 172.21.15.53:0/3256110666 >> v2:172.21.15.104:6816/32804 conn(0x7f9e4806e800 0x7f9e48070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e43fff640 1 -- 172.21.15.53:0/3256110666 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f9e4c08b160 con 0x7f9e5c0aca00 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e43fff640 10 client.4328.objecter ms_dispatch 0x7f9e5c000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e43fff640 3 client.4328.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e43fff640 3 client.4328.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e43fff640 20 client.4328.objecter dump_active .. 0 homeless 2024-09-15T02:43:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e5b7fe640 1 --2- 172.21.15.53:0/3256110666 >> v2:172.21.15.104:6816/32804 conn(0x7f9e4806e800 0x7f9e48070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.897+0000 7f9e5b7fe640 10 monclient: get_auth_request con 0x7f9e4806e800 auth_method 0 2024-09-15T02:43:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.898+0000 7f9e43fff640 10 client.4328.objecter ms_handle_connect 0x7f9e4806e800 2024-09-15T02:43:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.898+0000 7f9e5b7fe640 1 --2- 172.21.15.53:0/3256110666 >> v2:172.21.15.104:6816/32804 conn(0x7f9e4806e800 0x7f9e48070cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0b1740 tx=0x7f9e50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.901+0000 7f9e43fff640 1 -- 172.21.15.53:0/3256110666 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9e4c055840 con 0x7f9e5c0aca00 2024-09-15T02:43:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.901+0000 7f9e43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:25.901+0000 7f9e43fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:26.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.061+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e5c0b1f10 con 0x7f9e4806e800 2024-09-15T02:43:26.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.063+0000 7f9e43fff640 1 -- 172.21.15.53:0/3256110666 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48641 (secure 0 0 0) 0x7f9e5000b980 con 0x7f9e4806e800 2024-09-15T02:43:26.065 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:43:26.066 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:43:26.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 10 librados: watch_flush enter 2024-09-15T02:43:26.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 10 librados: watch_flush exit 2024-09-15T02:43:26.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 20 client.4328.objecter shutdown clearing up homeless session... 2024-09-15T02:43:26.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 10 client.4328.objecter successfully canceled tick 2024-09-15T02:43:26.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 >> v2:172.21.15.104:6816/32804 conn(0x7f9e4806e800 msgr2=0x7f9e48070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 --2- 172.21.15.53:0/3256110666 >> v2:172.21.15.104:6816/32804 conn(0x7f9e4806e800 0x7f9e48070cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0b1740 tx=0x7f9e50008040 comp rx=0 tx=0).stop 2024-09-15T02:43:26.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 10 monclient: shutdown 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 msgr2=0x7f9e5c0b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c0b1000 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c610 tx=0x7f9e4c007ed0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 shutdown_connections 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e5c0ae5e0 0x7f9e5c151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 --2- 172.21.15.53:0/3256110666 >> v2:172.21.15.104:6816/32804 conn(0x7f9e4806e800 0x7f9e48070cb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e5c0ae210 0x7f9e5c150f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 --2- 172.21.15.53:0/3256110666 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e5c0aca00 0x7f9e5c0b1000 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 >> 172.21.15.53:0/3256110666 conn(0x7f9e5c068590 msgr2=0x7f9e5c1427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:26.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 shutdown_connections 2024-09-15T02:43:26.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 -- 172.21.15.53:0/3256110666 wait complete. 2024-09-15T02:43:26.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.065+0000 7f9e62494640 1 librados: shutdown 2024-09-15T02:43:26.077 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-15T02:43:25.330543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2380192,"kb_used_data":1312,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752594528,"statfs":{"total":773094113280,"available":770656796672,"internally_reserved":0,"allocated":1343488,"data_stored":1444582,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.626528+0000","last_change":"2024-09-15T02:43:23.626724+0000","last_active":"2024-09-15T02:43:23.626528+0000","last_peered":"2024-09-15T02:43:23.626528+0000","last_clean":"2024-09-15T02:43:23.626528+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:43:23.626528+0000","last_undegraded":"2024-09-15T02:43:23.626528+0000","last_fullsized":"2024-09-15T02:43:23.626528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:48.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.425756+0000","last_change":"2024-09-15T02:43:23.426048+0000","last_active":"2024-09-15T02:43:23.425756+0000","last_peered":"2024-09-15T02:43:23.425756+0000","last_clean":"2024-09-15T02:43:23.425756+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:43:23.425756+0000","last_undegraded":"2024-09-15T02:43:23.425756+0000","last_fullsized":"2024-09-15T02:43:23.425756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:19.460873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.420262+0000","last_change":"2024-09-15T02:43:23.420483+0000","last_active":"2024-09-15T02:43:23.420262+0000","last_peered":"2024-09-15T02:43:23.420262+0000","last_clean":"2024-09-15T02:43:23.420262+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:43:23.420262+0000","last_undegraded":"2024-09-15T02:43:23.420262+0000","last_fullsized":"2024-09-15T02:43:23.420262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:03:14.233618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.425804+0000","last_change":"2024-09-15T02:43:23.426071+0000","last_active":"2024-09-15T02:43:23.425804+0000","last_peered":"2024-09-15T02:43:23.425804+0000","last_clean":"2024-09-15T02:43:23.425804+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:43:23.425804+0000","last_undegraded":"2024-09-15T02:43:23.425804+0000","last_fullsized":"2024-09-15T02:43:23.425804+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:00:03.954753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.423983+0000","last_change":"2024-09-15T02:43:23.423983+0000","last_active":"2024-09-15T02:43:23.423983+0000","last_peered":"2024-09-15T02:43:23.423983+0000","last_clean":"2024-09-15T02:43:23.423983+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:43:23.423983+0000","last_undegraded":"2024-09-15T02:43:23.423983+0000","last_fullsized":"2024-09-15T02:43:23.423983+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:45.683030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.419705+0000","last_change":"2024-09-15T02:43:23.419938+0000","last_active":"2024-09-15T02:43:23.419705+0000","last_peered":"2024-09-15T02:43:23.419705+0000","last_clean":"2024-09-15T02:43:23.419705+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:43:23.419705+0000","last_undegraded":"2024-09-15T02:43:23.419705+0000","last_fullsized":"2024-09-15T02:43:23.419705+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:09:35.753407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.420211+0000","last_change":"2024-09-15T02:43:23.420430+0000","last_active":"2024-09-15T02:43:23.420211+0000","last_peered":"2024-09-15T02:43:23.420211+0000","last_clean":"2024-09-15T02:43:23.420211+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:43:23.420211+0000","last_undegraded":"2024-09-15T02:43:23.420211+0000","last_fullsized":"2024-09-15T02:43:23.420211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:05:17.337850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T02:43:23.423233+0000","last_change":"2024-09-15T02:43:23.423477+0000","last_active":"2024-09-15T02:43:23.423233+0000","last_peered":"2024-09-15T02:43:23.423233+0000","last_clean":"2024-09-15T02:43:23.423233+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:43:23.423233+0000","last_undegraded":"2024-09-15T02:43:23.423233+0000","last_fullsized":"2024-09-15T02:43:23.423233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:05:30.605263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":101,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-15T02:43:22.410993+0000","last_change":"2024-09-15T02:43:20.387731+0000","last_active":"2024-09-15T02:43:22.410993+0000","last_peered":"2024-09-15T02:43:22.410993+0000","last_clean":"2024-09-15T02:43:22.410993+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:43:22.410993+0000","last_undegraded":"2024-09-15T02:43:22.410993+0000","last_fullsized":"2024-09-15T02:43:22.410993+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_clean_scrub_stamp":"2024-09-15T02:43:19.374225+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:33:29.816927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":212992,"data_stored":579614,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":147456,"data_stored":45986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":163840,"data_stored":50705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297500,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":147456,"data_stored":45986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":147456,"data_stored":45986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":50686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":163840,"data_stored":50705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":200704,"data_stored":574914,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:43:26.078 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T02:43:26.078 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T02:43:26.078 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T02:43:26.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T02:43:26.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.172+0000 7f928b15d640 10 monclient: get_monmap_and_config 2024-09-15T02:43:26.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.172+0000 7f928b15d640 10 monclient: build_initial_monmap 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.172+0000 7f928b15d640 10 monclient: monmap: 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:43:26.173245+0000 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:43:26.173245+0000 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:43:26.174 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 1 Processor -- start 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 1 -- start start 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient: init 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9284067510 addr v1:172.21.15.104:6789/0 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92840ab800 addr v1:172.21.15.53:6789/0 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9284141a10 0x7f92840ac120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient(hunting): picked mon.noname-a con 0x7f9284141a10 addr v2:172.21.15.53:3300/0 2024-09-15T02:43:26.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840678e0 con 0x7f92840ab800 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840acf10 con 0x7f9284067510 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9284140e00 con 0x7f9284141a10 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.173+0000 7f928b15d640 10 monclient(hunting): authenticate will time out at 742.743959s 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9284141a10 0x7f92840ac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9284141a10 0x7f92840ac120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51882/0 (socket says 172.21.15.53:51882) 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 1 -- 172.21.15.53:0/1534175323 learned_addr learned my addr 172.21.15.53:0/1534175323 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): get_auth_request con 0x7f9284141a10 auth_method 0 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f9283fff640 1 --1- 172.21.15.53:0/1534175323 >> v1:172.21.15.53:6789/0 conn(0x7f92840ab800 0x7f928413de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40756/0 (socket says 172.21.15.53:40756) 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 1 -- 172.21.15.53:0/1534175323 >> v1:172.21.15.104:6789/0 conn(0x7f9284067510 legacy=0x7f9284141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 1 -- 172.21.15.53:0/1534175323 >> v1:172.21.15.53:6789/0 conn(0x7f92840ab800 legacy=0x7f928413de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient: _finish_hunting 0 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 1 monclient: found mon.noname-a 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92896fa640 1 -- 172.21.15.53:0/1534175323 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928413d8f0 con 0x7f9284141a10 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92837fe640 1 -- 172.21.15.53:0/1534175323 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138021951 0 0) 0x7f92840678e0 con 0x7f92840ab800 2024-09-15T02:43:26.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.174+0000 7f92837fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92896fa640 1 --2- 172.21.15.53:0/1534175323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9284141a10 0x7f92840ac120 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f927c009910 tx=0x7f927c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108598fb23e6fc27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 1 -- 172.21.15.53:0/1534175323 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c02add0 con 0x7f9284141a10 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: dump: 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:26.185 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: _finish_auth 0 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 5 monclient: authenticate success, global_id 4371 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 1 -- 172.21.15.53:0/1534175323 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c032440 con 0x7f9284141a10 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 10 monclient: get_monmap_and_config success 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 4 set_mon_vals no callback set 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 1 -- 172.21.15.53:0/1534175323 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c0327e0 con 0x7f9284141a10 2024-09-15T02:43:26.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f92837fe640 10 monclient: dump: 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 10 monclient: shutdown 2024-09-15T02:43:26.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 1 -- 172.21.15.53:0/1534175323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9284141a10 msgr2=0x7f92840ac120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 1 --2- 172.21.15.53:0/1534175323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9284141a10 0x7f92840ac120 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f927c009910 tx=0x7f927c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 1 -- 172.21.15.53:0/1534175323 shutdown_connections 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 1 --2- 172.21.15.53:0/1534175323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9284141a10 0x7f92840ac120 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 1 -- 172.21.15.53:0/1534175323 >> 172.21.15.53:0/1534175323 conn(0x7f92840b07e0 msgr2=0x7f92840b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 1 -- 172.21.15.53:0/1534175323 shutdown_connections 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.175+0000 7f928b15d640 1 -- 172.21.15.53:0/1534175323 wait complete. 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 10 monclient: build_initial_monmap 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 10 monclient: monmap: 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:43:26.188 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 1 librados: starting msgr at 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 1 librados: starting objecter 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 1 Processor -- start 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 1 -- start start 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 1 librados: setting wanted keys 2024-09-15T02:43:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 1 librados: calling monclient init 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.176+0000 7f928b15d640 10 monclient: init 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient: _reopen_session rank -1 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9284067510 0x7f92841509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): picked mon.noname-c con 0x7f9284067510 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92840ab800 0x7f928406d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): picked mon.noname-a con 0x7f92840ab800 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9284141a10 0x7f928413d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): picked mon.noname-b con 0x7f9284141a10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928406dcb0 con 0x7f92840ab800 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:26.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928406caa0 con 0x7f9284141a10 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9284118230 con 0x7f9284067510 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): _renew_subs 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f928b15d640 10 monclient(hunting): authenticate will time out at 742.747529s 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92840ab800 0x7f928406d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9284067510 0x7f92841509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92840ab800 0x7f928406d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51896/0 (socket says 172.21.15.53:51896) 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 1 -- 172.21.15.53:0/3614281447 learned_addr learned my addr 172.21.15.53:0/3614281447 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): get_auth_request con 0x7f92840ab800 auth_method 0 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 10 monclient(hunting): get_auth_request con 0x7f9284067510 auth_method 0 2024-09-15T02:43:26.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9288ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f92896fa640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9284141a10 0x7f928413d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f92896fa640 10 monclient(hunting): get_auth_request con 0x7f9284141a10 auth_method 0 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f92896fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f92896fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:43:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f92896fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 1 -- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9284067510 msgr2=0x7f92841509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.177+0000 7f9283fff640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9284067510 0x7f92841509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 1 -- 172.21.15.53:0/3614281447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9284141a10 msgr2=0x7f928413d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9284141a10 0x7f928413d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 10 monclient: _finish_hunting 0 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 1 monclient: found mon.noname-a 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 1 -- 172.21.15.53:0/3614281447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c009450 con 0x7f92840ab800 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9288ef9640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9284067510 0x7f92841509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:43:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9283fff640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92840ab800 0x7f928406d670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f927400c960 tx=0x7f92740016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 client.?.objecter ms_handle_connect 0x7f92840ab800 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 1 -- 172.21.15.53:0/3614281447 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274005d60 con 0x7f92840ab800 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: dump: 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: _finish_auth 0 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 1 -- 172.21.15.53:0/3614281447 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927400a640 con 0x7f92840ab800 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 1 -- 172.21.15.53:0/3614281447 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400aa00 con 0x7f92840ab800 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 5 monclient: authenticate success, global_id 4377 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:43:26.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9280ff9640 10 monclient: dump: 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 10 monclient: _renew_subs 2024-09-15T02:43:26.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f9282ffd640 4 set_mon_vals no callback set 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928406e200 con 0x7f92840ab800 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 10 monclient: _renew_subs 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928406e700 con 0x7f92840ab800 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.178+0000 7f928b15d640 1 librados: init done 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.179+0000 7f928b15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.179+0000 7f928b15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.179+0000 7f928b15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.179+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928406e980 con 0x7f92840ab800 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.179+0000 7f9280ff9640 1 -- 172.21.15.53:0/3614281447 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f92740070f0 con 0x7f92840ab800 2024-09-15T02:43:26.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9280ff9640 1 --2- 172.21.15.53:0/3614281447 >> v2:172.21.15.104:6816/32804 conn(0x7f926406e800 0x7f9264070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9280ff9640 1 -- 172.21.15.53:0/3614281447 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4142+0+0 (secure 0 0 0) 0x7f927408b380 con 0x7f92840ab800 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9280ff9640 10 client.4377.objecter ms_dispatch 0x7f9284000b90 osd_map(18..18 src has 1..18) v4 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9280ff9640 3 client.4377.objecter handle_osd_map got epochs [18,18] > 0 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9280ff9640 3 client.4377.objecter handle_osd_map decoding full epoch 18 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9288ef9640 1 --2- 172.21.15.53:0/3614281447 >> v2:172.21.15.104:6816/32804 conn(0x7f926406e800 0x7f9264070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9288ef9640 10 monclient: get_auth_request con 0x7f926406e800 auth_method 0 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.180+0000 7f9280ff9640 20 client.4377.objecter dump_active .. 0 homeless 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.181+0000 7f9288ef9640 1 --2- 172.21.15.53:0/3614281447 >> v2:172.21.15.104:6816/32804 conn(0x7f926406e800 0x7f9264070cb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f926c007fd0 tx=0x7f926c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:43:26.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.181+0000 7f9280ff9640 10 client.4377.objecter ms_handle_connect 0x7f926406e800 2024-09-15T02:43:26.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.184+0000 7f9280ff9640 1 -- 172.21.15.53:0/3614281447 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9274055a30 con 0x7f92840ab800 2024-09-15T02:43:26.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.184+0000 7f9280ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:43:26.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.184+0000 7f9280ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:43:26.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.379+0000 7f928b15d640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-15T02:43:26.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.379+0000 7f928b15d640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-15T02:43:26.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.379+0000 7f928b15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:43:26.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.379+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f928406dfc0 con 0x7f92840ab800 2024-09-15T02:43:26.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.380+0000 7f9280ff9640 1 -- 172.21.15.53:0/3614281447 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f9274059240 con 0x7f92840ab800 2024-09-15T02:43:26.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.380+0000 7f9280ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-15T02:43:26.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.380+0000 7f9280ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:43:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:43:26.382 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T02:43:26.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 10 librados: watch_flush enter 2024-09-15T02:43:26.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 10 librados: watch_flush exit 2024-09-15T02:43:26.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 20 client.4377.objecter shutdown clearing up homeless session... 2024-09-15T02:43:26.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 10 client.4377.objecter successfully canceled tick 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 >> v2:172.21.15.104:6816/32804 conn(0x7f926406e800 msgr2=0x7f9264070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 --2- 172.21.15.53:0/3614281447 >> v2:172.21.15.104:6816/32804 conn(0x7f926406e800 0x7f9264070cb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f926c007fd0 tx=0x7f926c009040 comp rx=0 tx=0).stop 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 10 monclient: shutdown 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92840ab800 msgr2=0x7f928406d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92840ab800 0x7f928406d670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f927400c960 tx=0x7f92740016c0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 shutdown_connections 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 --2- 172.21.15.53:0/3614281447 >> v2:172.21.15.104:6816/32804 conn(0x7f926406e800 0x7f9264070cb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9284141a10 0x7f928413d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92840ab800 0x7f928406d670 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 --2- 172.21.15.53:0/3614281447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9284067510 0x7f92841509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 >> 172.21.15.53:0/3614281447 conn(0x7f92840b07e0 msgr2=0x7f928413abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:43:26.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 shutdown_connections 2024-09-15T02:43:26.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 -- 172.21.15.53:0/3614281447 wait complete. 2024-09-15T02:43:26.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:43:26.382+0000 7f928b15d640 1 librados: shutdown 2024-09-15T02:43:26.394 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T02:43:26.394 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T02:43:26.410 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:43:26.410 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T02:43:26.411 DEBUG:teuthology.orchestra.run.smithi053:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:43:26.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:43:26.427 INFO:teuthology.orchestra.run.smithi053.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T02:43:26.428 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:43:26.484 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T02:43:26.484 DEBUG:teuthology.orchestra.run.smithi053:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T02:43:26.539 INFO:tasks.workunit:timeout=6h 2024-09-15T02:43:26.540 INFO:tasks.workunit:cleanup=True 2024-09-15T02:43:26.540 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:43:26.596 INFO:tasks.workunit.client.0.smithi053.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T02:44:16.964 INFO:tasks.workunit.client.0.smithi053.stderr:Updating files: 97% (11736/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-15T02:44:18.075 INFO:tasks.workunit.client.0.smithi053.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-15T02:44:18.075 INFO:tasks.workunit.client.0.smithi053.stderr: 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr: 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr: 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr: git switch -c 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr: 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr:Or undo this operation with: 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr: 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr: git switch - 2024-09-15T02:44:18.076 INFO:tasks.workunit.client.0.smithi053.stderr: 2024-09-15T02:44:18.077 INFO:tasks.workunit.client.0.smithi053.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T02:44:18.077 INFO:tasks.workunit.client.0.smithi053.stderr: 2024-09-15T02:44:18.077 INFO:tasks.workunit.client.0.smithi053.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:44:18.081 DEBUG:teuthology.orchestra.run.smithi053:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T02:44:18.143 INFO:tasks.workunit.client.0.smithi053.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T02:44:18.145 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T02:44:18.145 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T02:44:18.749 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T02:44:18.818 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T02:44:18.859 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T02:44:18.860 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T02:44:18.860 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T02:44:18.899 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T02:44:18.902 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-15T02:44:18.902 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T02:44:18.959 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-15T02:44:18.961 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-15T02:44:18.961 DEBUG:teuthology.orchestra.run.smithi053:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-15T02:44:19.020 INFO:tasks.workunit.client.0.smithi053.stderr:+ parallel=1 2024-09-15T02:44:19.020 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' '' = --serial ']' 2024-09-15T02:44:19.020 INFO:tasks.workunit.client.0.smithi053.stderr:+ color= 2024-09-15T02:44:19.020 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -t 1 ']' 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:+ declare -A pids 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_aio 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_aio' 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_aio 2024-09-15T02:44:19.021 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.022 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_aio 2024-09-15T02:44:19.022 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37816 2024-09-15T02:44:19.022 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_aio on pid 37816' 2024-09-15T02:44:19.022 INFO:tasks.workunit.client.0.smithi053.stdout:test api_aio on pid 37816 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37816 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_aio_pp 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_aio_pp' 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_aio_pp 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.025 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.026 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.027 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.028 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.028 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.028 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.028 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_aio_pp 2024-09-15T02:44:19.028 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37823 2024-09-15T02:44:19.028 INFO:tasks.workunit.client.0.smithi053.stdout:test api_aio_pp on pid 37823 2024-09-15T02:44:19.031 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_aio_pp on pid 37823' 2024-09-15T02:44:19.031 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37823 2024-09-15T02:44:19.031 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_io 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_io' 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_io 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.032 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.033 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.033 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.033 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.033 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.033 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.033 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.034 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.035 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_io 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37832 2024-09-15T02:44:19.036 INFO:tasks.workunit.client.0.smithi053.stdout:test api_io on pid 37832 2024-09-15T02:44:19.043 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_io on pid 37832' 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37832 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_io_pp 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_io_pp' 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_io_pp 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.044 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.045 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.046 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.047 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_io_pp 2024-09-15T02:44:19.048 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37843 2024-09-15T02:44:19.049 INFO:tasks.workunit.client.0.smithi053.stdout:test api_io_pp on pid 37843 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_io_pp on pid 37843' 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37843 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_asio 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_asio' 2024-09-15T02:44:19.053 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_asio 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.054 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.055 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.056 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.057 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.058 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_asio 2024-09-15T02:44:19.059 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37858 2024-09-15T02:44:19.060 INFO:tasks.workunit.client.0.smithi053.stdout:test api_asio on pid 37858 2024-09-15T02:44:19.066 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_asio on pid 37858' 2024-09-15T02:44:19.066 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37858 2024-09-15T02:44:19.066 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.066 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_list 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_list' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_list 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.067 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.068 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.069 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.070 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.071 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.072 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.073 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.074 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37816/exe 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_list 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37875 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_list on pid 37875' 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-15T02:44:19.075 INFO:tasks.workunit.client.0.smithi053.stdout:test api_list on pid 37875 2024-09-15T02:44:19.082 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37875 2024-09-15T02:44:19.082 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.082 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.082 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.082 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.082 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_lock 2024-09-15T02:44:19.082 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_lock' 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_lock 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.083 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.084 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_aio 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.085 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.086 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.087 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.088 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.089 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_aio: /' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.090 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_lock 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37895 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_lock on pid 37895' 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.091 INFO:tasks.workunit.client.0.smithi053.stdout:test api_lock on pid 37895 2024-09-15T02:44:19.099 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37895 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37823/exe 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_lock_pp 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_lock_pp' 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.100 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_lock_pp 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.101 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.102 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.103 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.104 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.105 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.106 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.107 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37832/exe 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.108 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_aio_pp 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.109 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-15T02:44:19.110 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_lock_pp 2024-09-15T02:44:19.110 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37916 2024-09-15T02:44:19.110 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_lock_pp on pid 37916' 2024-09-15T02:44:19.110 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.110 INFO:tasks.workunit.client.0.smithi053.stdout:test api_lock_pp on pid 37916 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37916 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_misc 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.120 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_misc' 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.121 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_misc 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.122 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.123 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.124 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_io: /' 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.125 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.126 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_io.log 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_io 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37843/exe 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.127 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.128 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.129 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.130 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.131 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.132 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.132 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.132 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_misc 2024-09-15T02:44:19.132 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37939 2024-09-15T02:44:19.132 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_misc on pid 37939' 2024-09-15T02:44:19.132 INFO:tasks.workunit.client.0.smithi053.stdout:test api_misc on pid 37939 2024-09-15T02:44:19.140 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37939 2024-09-15T02:44:19.140 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_misc_pp 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_misc_pp' 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.141 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.142 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.143 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_misc_pp 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.144 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_io_pp 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37858/exe 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.145 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.146 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.147 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_misc_pp 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37957 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_misc_pp on pid 37957' 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.148 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.149 INFO:tasks.workunit.client.0.smithi053.stdout:test api_misc_pp on pid 37957 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37957 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.160 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.161 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.162 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_asio: /' 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_tier_pp 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.163 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_tier_pp' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.164 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.165 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.166 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.167 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_tier_pp 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.168 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_asio 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.169 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37895/exe 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_tier_pp 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.170 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37978 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.171 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_tier_pp on pid 37978' 2024-09-15T02:44:19.172 INFO:tasks.workunit.client.0.smithi053.stdout:test api_tier_pp on pid 37978 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=37978 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.187 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.188 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.189 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_lock 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.190 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.191 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.192 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.193 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_lock: /' 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_pool 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_pool' 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.194 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_pool 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37916/exe 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.197 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.198 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.199 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_lock_pp 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.200 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37875/exe 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_pool 2024-09-15T02:44:19.201 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38026 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.202 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.203 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.203 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_pool on pid 38026' 2024-09-15T02:44:19.203 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.203 INFO:tasks.workunit.client.0.smithi053.stdout:test api_pool on pid 38026 2024-09-15T02:44:19.219 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38026 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.220 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.221 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.222 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.223 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.224 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.225 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.226 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37939/exe 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_snapshots 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_snapshots' 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.227 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_snapshots 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_list.log 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.228 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_list 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_misc: /' 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37957/exe 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_list: /' 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_misc 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.229 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-15T02:44:19.230 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_misc_pp 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.231 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.232 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_snapshots 2024-09-15T02:44:19.233 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-15T02:44:19.234 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38090 2024-09-15T02:44:19.234 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-15T02:44:19.234 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_snapshots on pid 38090' 2024-09-15T02:44:19.234 INFO:tasks.workunit.client.0.smithi053.stdout:test api_snapshots on pid 38090 2024-09-15T02:44:19.237 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38090 2024-09-15T02:44:19.237 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.237 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.237 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.237 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_snapshots_pp 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_snapshots_pp' 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_snapshots_pp 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.238 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.239 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [----------] Global test environment set-up. 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-15T02:44:19.240 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.113+0000 7f67fc0c1600 10 monclient: get_monmap_and_config 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.113+0000 7f67fc0c1600 10 monclient: build_initial_monmap 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.113+0000 7f67fc0c1600 10 monclient: monmap: 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: epoch 0 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: last_changed 0.000000 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: created 0.000000 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: election_strategy: 1 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:44:19.241 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.113+0000 7f67fc0c1600 1 Processor -- start 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.113+0000 7f67fc0c1600 1 -- start start 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.113+0000 7f67fc0c1600 10 monclient: init 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.115+0000 7f67fc0c1600 10 monclient: _reopen_session rank -1 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.115+0000 7f67fc0c1600 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.115+0000 7f67fc0c1600 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b3710 0x5566f19e71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.115+0000 7f67f864a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b3710 0x5566f19e71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.115+0000 7f67f864a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b3710 0x5566f19e71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:33822/0 (socket says 172.21.15.53:33822) 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.116+0000 7f67f864a640 1 -- 172.21.15.53:0/526963807 learned_addr learned my addr 172.21.15.53:0/526963807 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:44:19.242 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.116+0000 7f67fc0c1600 10 monclient(hunting): picked mon.noname-c con 0x5566f19b3710 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:44:19.243 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.116+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b40c0 0x5566f1a78dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:19.243 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.116+0000 7f67fc0c1600 10 monclient(hunting): picked mon.noname-b con 0x5566f19b40c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:44:19.243 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.116+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f1a7a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:19.243 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.116+0000 7f67f8e4b640 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f1a7a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.245 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.246 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.116+0000 7f67f7e49640 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b40c0 0x5566f1a78dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 10 monclient(hunting): picked mon.noname-a con 0x5566f1a7a490 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 10 monclient(hunting): start opening mon connection 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f1926b30 con 0x5566f1a7a490 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 10 monclient(hunting): start opening mon connection 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f19267d0 con 0x5566f19b40c0 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 10 monclient(hunting): start opening mon connection 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566f18e9730 con 0x5566f19b3710 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 10 monclient(hunting): _renew_subs 2024-09-15T02:44:19.247 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67fc0c1600 10 monclient(hunting): authenticate will time out at 795.687150s 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f864a640 10 monclient(hunting): get_auth_request con 0x5566f19b3710 auth_method 0 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f864a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f864a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f864a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f8e4b640 10 monclient(hunting): get_auth_request con 0x5566f1a7a490 auth_method 0 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f8e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f8e4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f8e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f8e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f8e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f8e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:44:19.248 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f864a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:44:19.249 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f864a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:44:19.249 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.117+0000 7f67f864a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:44:19.249 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 10 monclient(hunting): handle_auth_done global_id 4395 payload 274 2024-09-15T02:44:19.249 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 1 -- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b3710 msgr2=0x5566f19e71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:44:19.249 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b3710 0x5566f19e71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:19.250 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_snapshots_pp 2024-09-15T02:44:19.250 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38125 2024-09-15T02:44:19.250 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_snapshots_pp on pid 38125' 2024-09-15T02:44:19.250 INFO:tasks.workunit.client.0.smithi053.stdout: test api_snapshots_pp on pid 38125 2024-09-15T02:44:19.258 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.258 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38125 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_stat 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.259 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_stat' 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.260 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.261 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_stat 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.262 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.263 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.264 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_stat 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38146 2024-09-15T02:44:19.265 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_stat on pid 38146' 2024-09-15T02:44:19.266 INFO:tasks.workunit.client.0.smithi053.stdout:test api_stat on pid 38146 2024-09-15T02:44:19.269 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38146 2024-09-15T02:44:19.269 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.269 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.269 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_stat_pp 2024-09-15T02:44:19.269 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_stat_pp' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/37978/exe 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_stat_pp 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.270 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_stat_pp 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38162 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.272 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_stat_pp on pid 38162' 2024-09-15T02:44:19.273 INFO:tasks.workunit.client.0.smithi053.stdout:test api_stat_pp on pid 38162 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38162 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_watch_notify 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_tier_pp 2024-09-15T02:44:19.291 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_watch_notify' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.292 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.293 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.294 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.295 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.296 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.297 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_watch_notify 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.298 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.299 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.300 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.301 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38026/exe 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.302 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.303 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.304 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_pool 2024-09-15T02:44:19.305 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_watch_notify 2024-09-15T02:44:19.306 INFO:tasks.workunit.client.0.smithi053.stdout:test api_watch_notify on pid 38223 2024-09-15T02:44:19.311 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38223 2024-09-15T02:44:19.311 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-15T02:44:19.311 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_watch_notify on pid 38223' 2024-09-15T02:44:19.311 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38223 2024-09-15T02:44:19.311 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.311 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_pool: /' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.312 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_watch_notify_pp 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_watch_notify_pp' 2024-09-15T02:44:19.313 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.314 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.315 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_watch_notify_pp 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_watch_notify_pp 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38251 2024-09-15T02:44:19.316 INFO:tasks.workunit.client.0.smithi053.stdout:test api_watch_notify_pp on pid 38251 2024-09-15T02:44:19.330 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_watch_notify_pp on pid 38251' 2024-09-15T02:44:19.330 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38251 2024-09-15T02:44:19.330 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.330 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.330 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.330 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.330 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.331 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.332 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_cmd 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_cmd' 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38090/exe 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.333 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.334 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.335 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.336 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_cmd 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.337 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.338 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38125/exe 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_snapshots 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.339 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.340 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.341 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.342 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_snapshots_pp 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.343 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.344 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.344 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.344 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.344 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-15T02:44:19.344 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_cmd 2024-09-15T02:44:19.344 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38299 2024-09-15T02:44:19.344 INFO:tasks.workunit.client.0.smithi053.stdout:test api_cmd on pid 38299 2024-09-15T02:44:19.359 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_cmd on pid 38299' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38299 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_cmd_pp 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_cmd_pp' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.360 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.361 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.362 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.363 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.364 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38162/exe 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.365 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.366 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.367 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.368 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.369 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_cmd_pp 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.370 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.371 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.372 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_cmd_pp 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_stat_pp 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38146/exe 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38338 2024-09-15T02:44:19.373 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.374 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.374 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.374 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_cmd_pp on pid 38338' 2024-09-15T02:44:19.374 INFO:tasks.workunit.client.0.smithi053.stdout:test api_cmd_pp on pid 38338 2024-09-15T02:44:19.382 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38338 2024-09-15T02:44:19.382 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.382 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.382 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.382 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_service 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_service' 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.383 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.384 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_stat 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.385 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.386 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_service 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.387 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.388 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.389 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_service 2024-09-15T02:44:19.390 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38358 2024-09-15T02:44:19.390 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_service on pid 38358' 2024-09-15T02:44:19.390 INFO:tasks.workunit.client.0.smithi053.stdout:test api_service on pid 38358 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38358 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.400 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.401 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.402 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_service_pp 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.403 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_stat: /' 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_service_pp' 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.404 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_service_pp 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38223/exe 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.405 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.406 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.407 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_watch_notify 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38251/exe 2024-09-15T02:44:19.408 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.409 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 1 -- 172.21.15.53:0/526963807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b40c0 msgr2=0x5566f1a78dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b40c0 0x5566f1a78dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 10 monclient: _finish_hunting 0 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 1 monclient: found mon.noname-a 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f8e4b640 1 -- 172.21.15.53:0/526963807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f000a180 con 0x5566f1a7a490 2024-09-15T02:44:19.410 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.118+0000 7f67f864a640 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b3710 0x5566f19e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.119+0000 7f67f8e4b640 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f1a7a860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00a5b0 tx=0x7f67ec005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c171d5a32b48c99 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.119+0000 7f67f7e49640 10 monclient: get_auth_request con 0x5566f19b40c0 auth_method 0 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.119+0000 7f67f7e49640 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b40c0 0x5566f1a78dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67f6e47640 1 -- 172.21.15.53:0/526963807 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec00c4d0 con 0x5566f1a7a490 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67f6e47640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67f6e47640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67f6e47640 10 monclient: dump: 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: epoch 1 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: election_strategy: 1 2024-09-15T02:44:19.411 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:44:19.412 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:44:19.412 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:44:19.412 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.412 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67f6e47640 10 monclient: _finish_auth 0 2024-09-15T02:44:19.412 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67f6e47640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:44:19.412 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67fc0c1600 5 monclient: authenticate success, global_id 4395 2024-09-15T02:44:19.412 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.182+0000 7f67fc0c1600 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_service_pp 2024-09-15T02:44:19.413 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67f6e47640 1 -- 172.21.15.53:0/526963807 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec00c630 con 0x5566f1a7a490 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67f6e47640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67f6e47640 1 -- 172.21.15.53:0/526963807 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec00ca10 con 0x5566f1a7a490 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67f6e47640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67f6e47640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67f6e47640 10 monclient: dump: 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: epoch 1 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: election_strategy: 1 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:44:19.414 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67fc0c1600 10 monclient: get_monmap_and_config success 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.219+0000 7f67fc0c1600 4 set_mon_vals no callback set 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.220+0000 7f67fc0c1600 10 monclient: shutdown 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.220+0000 7f67fc0c1600 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.220+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 msgr2=0x5566f1a7a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.220+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f1a7a860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00a5b0 tx=0x7f67ec005910 comp rx=0 tx=0).stop 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 shutdown_connections 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f1a7a860 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b40c0 0x5566f1a78dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/526963807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b3710 0x5566f19e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:19.415 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 >> 172.21.15.53:0/526963807 conn(0x5566f19f2500 msgr2=0x5566f19ed230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 shutdown_connections 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 1 -- 172.21.15.53:0/526963807 wait complete. 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 10 monclient: build_initial_monmap 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.221+0000 7f67fc0c1600 10 monclient: monmap: 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: epoch 0 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38401 2024-09-15T02:44:19.416 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_service_pp on pid 38401' 2024-09-15T02:44:19.417 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio:test api_service_pp on pid 38401 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38401 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.418 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: last_changed 0.000000 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: created 0.000000 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: election_strategy: 1 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:44:19.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.222+0000 7f67fc0c1600 1 librados: starting msgr at 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.222+0000 7f67fc0c1600 1 librados: starting objecter 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.222+0000 7f67fc0c1600 1 Processor -- start 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.222+0000 7f67fc0c1600 1 -- start start 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.222+0000 7f67fc0c1600 1 librados: setting wanted keys 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.222+0000 7f67fc0c1600 1 librados: calling monclient init 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.222+0000 7f67fc0c1600 10 monclient: init 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient: _reopen_session rank -1 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b3710 0x5566f1a6dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:19.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): picked mon.noname-b con 0x5566f19b3710 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b40c0 0x5566f19a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): picked mon.noname-c con 0x5566f19b40c0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f19ea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): picked mon.noname-a con 0x5566f1a7a490 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): start opening mon connection 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f19eb010 con 0x5566f1a7a490 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): start opening mon connection 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f1a6fbb0 con 0x5566f19b3710 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): start opening mon connection 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566f1a6fe20 con 0x5566f19b40c0 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): _renew_subs 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67fc0c1600 10 monclient(hunting): authenticate will time out at 795.793338s 2024-09-15T02:44:19.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f7e49640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b40c0 0x5566f19a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f7e49640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b40c0 0x5566f19a5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:33900/0 (socket says 172.21.15.53:33900) 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b3710 0x5566f1a6dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f7e49640 1 -- 172.21.15.53:0/2900149002 learned_addr learned my addr 172.21.15.53:0/2900149002 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f7e49640 10 monclient(hunting): get_auth_request con 0x5566f19b40c0 auth_method 0 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f7e49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f7e49640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f7e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 10 monclient(hunting): get_auth_request con 0x5566f19b3710 auth_method 0 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:44:19.422 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f864a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.223+0000 7f67f8e4b640 1 --2- 172.21.15.53:0/2900149002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f19ea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f8e4b640 10 monclient(hunting): get_auth_request con 0x5566f1a7a490 auth_method 0 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f8e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f8e4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f8e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 1 -- 172.21.15.53:0/2900149002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b40c0 msgr2=0x5566f19a5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 1 --2- 172.21.15.53:0/2900149002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b40c0 0x5566f19a5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:19.423 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 1 -- 172.21.15.53:0/2900149002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 msgr2=0x5566f19ea2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:44:19.428 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.428 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.429 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.430 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_c_write_operations 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38299/exe 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_c_write_operations' 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_c_write_operations 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.432 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 1 --2- 172.21.15.53:0/2900149002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5566f1a7a490 0x5566f19ea2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 10 monclient: _finish_hunting 0 2024-09-15T02:44:19.433 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 1 monclient: found mon.noname-b 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3300/0 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 1 -- 172.21.15.53:0/2900149002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e8005d20 con 0x5566f19b3710 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f7e49640 1 --2- 172.21.15.53:0/2900149002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5566f19b40c0 0x5566f19a5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.224+0000 7f67f864a640 1 --2- 172.21.15.53:0/2900149002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b3710 0x5566f1a6dda0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f67e800bfd0 tx=0x7f67e8005a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 10 client.?.objecter ms_handle_connect 0x5566f19b3710 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 10 client.?.objecter resend_mon_ops 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e8010070 con 0x5566f19b3710 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:44:19.434 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 10 monclient: dump: 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: epoch 1 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: election_strategy: 1 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.435 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 10 monclient: _finish_auth 0 2024-09-15T02:44:19.436 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:44:19.436 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e8008970 con 0x5566f19b3710 2024-09-15T02:44:19.436 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f5644640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:44:19.436 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.229+0000 7f67f6e47640 4 set_mon_vals no callback set 2024-09-15T02:44:19.436 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 5 monclient: authenticate success, global_id 4423 2024-09-15T02:44:19.436 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 10 monclient: _renew_subs 2024-09-15T02:44:19.436 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:44:19.438 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-15T02:44:19.438 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_cmd: /' 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.439 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.440 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec005bd0 con 0x5566f19b3710 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 10 monclient: _renew_subs 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566f1915070 con 0x5566f19b3710 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 1 librados: init done 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67fc0c1600 10 librados: wait_for_osdmap waiting 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e8008cf0 con 0x5566f19b3710 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.230+0000 7f67f5644640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:44:19.441 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.231+0000 7f67f5644640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.231+0000 7f67f5644640 10 monclient: dump: 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: epoch 1 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: election_strategy: 1 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.234+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f67e8014620 con 0x5566f19b3710 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.234+0000 7f67f5644640 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6816/32804 conn(0x7f67e406cb30 0x7f67e406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:19.442 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.234+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4114+0+0 (secure 0 0 0) 0x7f67e808a170 con 0x5566f19b3710 2024-09-15T02:44:19.443 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.234+0000 7f67f5644640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_map(19..19 src has 1..19) v4 2024-09-15T02:44:19.443 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.234+0000 7f67f5644640 3 client.4423.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T02:44:19.443 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.234+0000 7f67f5644640 3 client.4423.objecter handle_osd_map decoding full epoch 19 2024-09-15T02:44:19.443 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67f5644640 20 client.4423.objecter dump_active .. 0 homeless 2024-09-15T02:44:19.443 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67fc0c1600 10 librados: wait_for_osdmap done waiting 2024-09-15T02:44:19.443 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67fc0c1600 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:44:19.443 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f67f00519b0 con 0x5566f19b3710 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.447 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.448 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.449 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_c_write_operations 2024-09-15T02:44:19.450 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38456 2024-09-15T02:44:19.451 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67f7e49640 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6816/32804 conn(0x7f67e406cb30 0x7f67e406efe0 untest api_c_write_operations on pid 38456 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_c_write_operations on pid 38456' 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38456 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_cmd 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.462 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.463 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.464 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.465 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.466 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_c_read_operations 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_c_read_operations' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.467 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.468 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_c_read_operations 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.469 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.470 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.471 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.472 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.472 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.472 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.472 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_c_read_operations 2024-09-15T02:44:19.472 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38504 2024-09-15T02:44:19.472 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_c_read_operations on pid 38504' 2024-09-15T02:44:19.472 INFO:tasks.workunit.client.0.smithi053.stdout:test api_c_read_operations on pid 38504 2024-09-15T02:44:19.479 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38504 2024-09-15T02:44:19.479 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.479 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.479 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38358/exe 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s api_cls_remote_reads 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' api_cls_remote_reads' 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.480 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38338/exe 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo api_cls_remote_reads 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.481 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.482 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.484 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.484 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.484 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.484 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.484 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.484 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.485 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.486 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.487 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=api_cls_remote_reads 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_cmd_pp 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38531 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test api_cls_remote_reads on pid 38531' 2024-09-15T02:44:19.488 INFO:tasks.workunit.client.0.smithi053.stdout:test api_cls_remote_reads on pid 38531 2024-09-15T02:44:19.494 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38531 2024-09-15T02:44:19.494 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.494 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.494 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-15T02:44:19.494 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_service 2024-09-15T02:44:19.494 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s list_parallel 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' list_parallel' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo list_parallel 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_service.log 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_service: /' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.495 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.496 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.497 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.498 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=list_parallel 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.499 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.500 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.500 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.500 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38559 2024-09-15T02:44:19.500 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test list_parallel on pid 38559' 2024-09-15T02:44:19.500 INFO:tasks.workunit.client.0.smithi053.stdout:test list_parallel on pid 38559 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38559 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s open_pools_parallel 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.509 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.510 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' open_pools_parallel' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.511 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.512 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.513 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo open_pools_parallel 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.514 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.515 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38401/exe 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.516 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.517 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=open_pools_parallel 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38594 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test open_pools_parallel on pid 38594' 2024-09-15T02:44:19.518 INFO:tasks.workunit.client.0.smithi053.stdout:test open_pools_parallel on pid 38594 2024-09-15T02:44:19.530 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38594 2024-09-15T02:44:19.530 INFO:tasks.workunit.client.0.smithi053.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T02:44:19.530 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.530 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.530 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.531 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.532 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_service_pp 2024-09-15T02:44:19.533 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ printf %25s delete_pools_parallel 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:+ r=' delete_pools_parallel' 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.534 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.535 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38456/exe 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:++ awk '{print $1}' 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:++ echo delete_pools_parallel 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.536 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_c_write_operations 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.537 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.538 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.539 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.540 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-15T02:44:19.541 INFO:tasks.workunit.client.0.smithi053.stdout:test delete_pools_parallel on pid 38646 2024-09-15T02:44:19.567 INFO:tasks.workunit.client.0.smithi053.stderr:+ ff=delete_pools_parallel 2024-09-15T02:44:19.567 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38646 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ echo 'test delete_pools_parallel on pid 38646' 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ pids[$f]=38646 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ ret=0 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37978 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37978 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.568 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.569 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.570 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -f /etc/bashrc ']' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:+ . /etc/bashrc 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -z '' ']' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ BASHRCSOURCED=Y 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ shopt -q login_shell 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.572 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ umask 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' 0022 -eq 0 ']' 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.573 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ SHELL=/bin/bash 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.574 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.575 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.576 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38504/exe 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38531/exe 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.577 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.578 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.579 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-15T02:44:19.580 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.581 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'grep=grep --color=auto' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-15T02:44:19.582 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorls.sh 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' '!' -t 0 ']' 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:+++ return 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export S_COLORS=auto 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:+++ S_COLORS=auto 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.583 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.584 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.585 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.586 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_api_c_read_operations 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.587 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38559/exe 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.588 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ list_parallel: /' 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-15T02:44:19.589 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_list_parallel 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.590 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38594/exe 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/gawk.sh 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/lang.sh 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup 2024-09-15T02:44:19.591 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/locale 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.592 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_open_pools_parallel 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:++++ export LANG=en_US.UTF-8 2024-09-15T02:44:19.593 INFO:tasks.workunit.client.0.smithi053.stderr:++++ LANG=en_US.UTF-8 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ LANG=en_US.UTF-8 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ unset LANG_backup config 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n '' ']' 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' dumb = linux ']' 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/less.sh 2024-09-15T02:44:19.594 INFO:tasks.workunit.client.0.smithi053.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:++ for i in /etc/profile.d/*.sh 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:++ '[' '' ']' 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:++ . /etc/profile.d/which2.sh 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:+++++ readlink /proc/38646/exe 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:++++ basename /usr/bin/bash 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_declare='declare -f' 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:+++ which_opt=-f 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export which_declare 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:+++ export -f which 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset i 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:++ unset -f pathmunge 2024-09-15T02:44:19.595 INFO:tasks.workunit.client.0.smithi053.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-15T02:44:19.596 INFO:tasks.workunit.client.0.smithi053.stderr:+ export PATH 2024-09-15T02:44:19.596 INFO:tasks.workunit.client.0.smithi053.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-15T02:44:19.596 INFO:tasks.workunit.client.0.smithi053.stderr:+ unset rc 2024-09-15T02:44:19.596 INFO:tasks.workunit.client.0.smithi053.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-15T02:44:19.596 INFO:tasks.workunit.client.0.smithi053.stderr:+ ceph_test_rados_delete_pools_parallel 2024-09-15T02:44:19.596 INFO:tasks.workunit.client.0.smithi053.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-15T02:44:20.453 INFO:tasks.workunit.client.0.smithi053.stdout:known :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:20.453 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67f7e49640 10 monclient: get_auth_request con 0x7f67e406cb30 auth_method 0 2024-09-15T02:44:20.453 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67e8054840 con 0x5566f19b3710 2024-09-15T02:44:20.453 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67f5644640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T02:44:20.453 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67f6646640 10 client.4423.objecter _get_latest_version latest 19, have it 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67fc0c1600 10 client.4423.objecter create_pool name=ceph_test_rados_api_asio 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67fc0c1600 10 client.4423.objecter pool_op_submit 1 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67fc0c1600 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.235+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x5566f1967ec0 con 0x5566f19b3710 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.236+0000 7f67f7e49640 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6816/32804 conn(0x7f67e406cb30 0x7f67e406efe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x5566f19a6400 tx=0x7f67f0057000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:19.236+0000 7f67f5644640 10 client.4423.objecter ms_handle_connect 0x7f67e406cb30 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v21) v1 ==== 43+0+0 (secure 0 0 0) 0x7f67e8019030 con 0x5566f19b3710 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 pool_op_reply(tid 1 (0) Success v21) v1 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 10 client.4423.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v21) v1 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 10 client.4423.objecter have request 1 at 0x5566f1a6f1c0 Op: create 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 20 client.4423.objecter waiting for client to reach epoch 21 before calling back 2024-09-15T02:44:20.454 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:44:20.455 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 10 monclient: _renew_subs 2024-09-15T02:44:20.455 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.104:3300/0 2024-09-15T02:44:20.455 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=20}) v3 -- 0x7f67e40723f0 con 0x5566f19b3710 2024-09-15T02:44:20.455 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.444+0000 7f67f5644640 10 client.4423.objecter done 2024-09-15T02:44:20.455 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.447+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== mon.1 v2:172.21.15.104:3300/0 8 ==== osd_map(20..21 src has 1..21) v4 ==== 12507+0+0 (secure 0 0 0) 0x7f67e80583f0 con 0x5566f19b3710 2024-09-15T02:44:20.455 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.447+0000 7f67f5644640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_map(20..21 src has 1..21) v4 2024-09-15T02:44:20.455 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.447+0000 7f67f5644640 3 client.4423.objecter handle_osd_map got epochs [20,21] > 19 2024-09-15T02:44:21.596 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.447+0000 7f67f5644640 3 client.4423.objecter handle_osd_map decoding incremental epoch 20 2024-09-15T02:44:21.596 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.447+0000 7f67f5644640 3 client.4423.objecter handle_osd_map decoding incremental epoch 21 2024-09-15T02:44:21.596 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.458+0000 7f67f5644640 20 client.4423.objecter dump_active .. 0 homeless 2024-09-15T02:44:21.596 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 10 librados: set snap read head -> 1 2024-09-15T02:44:21.596 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 10 librados: writefull oid=exist nspace= 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f19ea810 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 21 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6808/33357 conn(0x5566f1a6d330 0x5566f1aa9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 2 osd.6 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 2 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 15 client.4423.objecter _send_op 2 to 12.1 on osd.6 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) v8 -- 0x5566f1aa9750 con 0x5566f1a6d330 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.597 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.459+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.495+0000 7f67f8e4b640 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6808/33357 conn(0x5566f1a6d330 0x5566f1aa9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.495+0000 7f67f8e4b640 10 monclient: get_auth_request con 0x5566f1a6d330 auth_method 0 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.515+0000 7f67f8e4b640 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6808/33357 conn(0x5566f1a6d330 0x5566f1aa9210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:20.515+0000 7f67f5644640 10 client.4423.objecter ms_handle_connect 0x5566f1a6d330 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 1 ==== osd_map(22..22 src has 1..22) v4 ==== 7285+0+0 (crc 0 0 0) 0x7f67ec019d30 con 0x5566f1a6d330 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_map(22..22 src has 1..22) v4 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 3 client.4423.objecter handle_osd_map got epochs [22,22] > 21 2024-09-15T02:44:21.598 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 3 client.4423.objecter handle_osd_map decoding incremental epoch 22 2024-09-15T02:44:21.625 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 10 client.4423.objecter checking op 2 2024-09-15T02:44:21.625 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 20 client.4423.objecter dump_active .. 0 homeless 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.595+0000 7f67f5644640 20 client.4423.objecter 2 12.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 2 ==== osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) v8 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 2 ondisk uv 1 in 12.1 attempt 0 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 2 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 2 2024-09-15T02:44:21.626 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 2 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 10 librados: Objecter returned from writefull r=0 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f19ea810 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 3 osd.6 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 3 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 15 client.4423.objecter _send_op 3 to 12.1 on osd.6 2024-09-15T02:44:21.627 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:3 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f19e81c0 con 0x5566f1a6d330 2024-09-15T02:44:21.628 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.628 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.628 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f19e85b0 2024-09-15T02:44:21.628 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.628 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 4 osd.6 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 4 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 15 client.4423.objecter _send_op 4 to 12.1 on osd.6 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:4 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f19e8aa0 con 0x5566f1a6d330 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.624+0000 7f67fc0c1600 5 client.4423.objecter 2 in flight 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.629 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-15T02:44:21.630 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.630 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 3 ondisk uv 1 in 12.1 attempt 0 2024-09-15T02:44:21.630 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 5 2024-09-15T02:44:21.630 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 3 2024-09-15T02:44:21.631 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 3 2024-09-15T02:44:21.631 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.631 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 3 2024-09-15T02:44:21.631 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.631 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.631 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 4 ondisk uv 0 in 12.1 attempt 0 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 4 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 4 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 4 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f19e85b0 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.632 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 5 osd.6 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 5 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 15 client.4423.objecter _send_op 5 to 12.1 on osd.6 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:5 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f19ea810 con 0x5566f1a6d330 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aa9b00 2024-09-15T02:44:21.633 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.634 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-09-15T02:44:21.634 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.634 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.634 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 6 osd.6 2024-09-15T02:44:21.634 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.634 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 6 2024-09-15T02:44:21.634 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 15 client.4423.objecter _send_op 6 to 12.1 on osd.6 2024-09-15T02:44:21.635 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:6 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aa9ff0 con 0x5566f1a6d330 2024-09-15T02:44:21.635 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.635 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.626+0000 7f67fc0c1600 5 client.4423.objecter 2 in flight 2024-09-15T02:44:21.635 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (3 ms) 2024-09-15T02:44:21.635 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-15T02:44:21.635 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.635 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-15T02:44:21.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 5 ondisk uv 1 in 12.1 attempt 0 2024-09-15T02:44:21.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 5 2024-09-15T02:44:21.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 5 2024-09-15T02:44:21.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 5 2024-09-15T02:44:21.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 5 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 6 ondisk uv 0 in 12.1 attempt 0 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 6 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 6 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 6 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aa9b00 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 7 osd.6 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.638 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 7 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 15 client.4423.objecter _send_op 7 to 12.1 on osd.6 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:7 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f19e8590 con 0x5566f1a6d330 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1acaa30 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 8 osd.6 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 8 2024-09-15T02:44:21.639 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 15 client.4423.objecter _send_op 8 to 12.1 on osd.6 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:8 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1acaf20 con 0x5566f1a6d330 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.629+0000 7f67fc0c1600 5 client.4423.objecter 2 in flight 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 7 ondisk uv 1 in 12.1 attempt 0 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 5 2024-09-15T02:44:21.640 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 7 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 7 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 7 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.630+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 8 ondisk uv 0 in 12.1 attempt 0 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:44:21.641 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 8 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 8 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 8 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 librados: queue_aio_write 0x5566f195cce0 completion 0x5566f19102d0 write_seq 1 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aa9b00 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.642 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 9 osd.6 2024-09-15T02:44:21.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 9 2024-09-15T02:44:21.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 15 client.4423.objecter _send_op 9 to 12.1 on osd.6 2024-09-15T02:44:21.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:9 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aab3b0 con 0x5566f1a6d330 2024-09-15T02:44:21.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.631+0000 7f67fc0c1600 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T02:44:21.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (5 ms) 2024-09-15T02:44:21.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-15T02:44:21.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 9 ==== osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) v8 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 9 ondisk uv 2 in 12.1 attempt 0 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 9 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 9 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 9 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67f8e4b640 20 librados: complete_aio_write 0x5566f19102d0 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 librados: queue_aio_write 0x5566f195cce0 completion 0x5566f1aab760 write_seq 2 2024-09-15T02:44:21.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aa9b00 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 10 osd.6 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 10 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 15 client.4423.objecter _send_op 10 to 12.1 on osd.6 2024-09-15T02:44:21.646 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:10 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aab3b0 con 0x5566f1a6d330 2024-09-15T02:44:21.647 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.647 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.647 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.635+0000 7f67fc0c1600 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T02:44:21.647 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (8 ms) 2024-09-15T02:44:21.648 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-15T02:44:21.648 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 10 ==== osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.648 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) v8 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 10 ondisk uv 3 in 12.1 attempt 0 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 10 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 10 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 10 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67f8e4b640 20 librados: complete_aio_write 0x5566f1aab760 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 librados: queue_aio_write 0x5566f195cce0 completion 0x5566f1a77720 write_seq 3 2024-09-15T02:44:21.649 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aa9b00 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 11 osd.6 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 11 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 15 client.4423.objecter _send_op 11 to 12.1 on osd.6 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:11 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aab3b0 con 0x5566f1a6d330 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.650 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.643+0000 7f67fc0c1600 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4 ms) 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 11 ==== osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) v8 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 11 ondisk uv 4 in 12.1 attempt 0 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 11 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 11 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 11 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.651 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67f8e4b640 20 librados: complete_aio_write 0x5566f1a77720 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aaa3a0 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 12 osd.6 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 12 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 15 client.4423.objecter _send_op 12 to 12.1 on osd.6 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:12 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aaa890 con 0x5566f1a6d330 2024-09-15T02:44:21.652 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.653 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.653 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aaad50 2024-09-15T02:44:21.653 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.653 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-09-15T02:44:21.653 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.653 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.653 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 13 osd.6 2024-09-15T02:44:21.654 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.654 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 13 2024-09-15T02:44:21.654 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 15 client.4423.objecter _send_op 13 to 12.1 on osd.6 2024-09-15T02:44:21.654 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:13 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aab240 con 0x5566f1a6d330 2024-09-15T02:44:21.654 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.654 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.647+0000 7f67fc0c1600 5 client.4423.objecter 2 in flight 2024-09-15T02:44:21.654 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 12 ondisk uv 4 in 12.1 attempt 0 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 5 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 12 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 12 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 12 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-15T02:44:21.655 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 13 ondisk uv 0 in 12.1 attempt 0 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 13 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 13 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 13 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.656 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aaa3a0 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 14 osd.6 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 14 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 15 client.4423.objecter _send_op 14 to 12.1 on osd.6 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:14 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aaa890 con 0x5566f1a6d330 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aaad50 2024-09-15T02:44:21.657 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 15 osd.6 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 15 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 15 client.4423.objecter _send_op 15 to 12.1 on osd.6 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:15 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aab240 con 0x5566f1a6d330 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.648+0000 7f67fc0c1600 5 client.4423.objecter 2 in flight 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (2 ms) 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-15T02:44:21.658 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 14 ondisk uv 4 in 12.1 attempt 0 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 5 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 14 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 14 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 14 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.659 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 15 ondisk uv 0 in 12.1 attempt 0 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 15 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 15 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 15 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1abd460 2024-09-15T02:44:21.660 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 16 osd.6 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 16 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 15 client.4423.objecter _send_op 16 to 12.1 on osd.6 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:16 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aaad50 con 0x5566f1a6d330 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1abdbe0 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T02:44:21.661 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 17 osd.6 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 17 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 15 client.4423.objecter _send_op 17 to 12.1 on osd.6 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:17 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1abe0d0 con 0x5566f1a6d330 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67fc0c1600 5 client.4423.objecter 2 in flight 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.662 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.649+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 16 ondisk uv 4 in 12.1 attempt 0 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 5 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 16 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 16 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 5 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 16 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T02:44:21.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 17 ondisk uv 0 in 12.1 attempt 0 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 17 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 17 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 17 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 20 librados: queue_aio_write 0x5566f195cce0 completion 0x5566f1aab3c0 write_seq 4 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aaa3a0 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 18 osd.6 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 18 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 15 client.4423.objecter _send_op 18 to 12.1 on osd.6 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:18 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aaa890 con 0x5566f1a6d330 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.650+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 18 ==== osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) v8 2024-09-15T02:44:21.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 18 ondisk uv 5 in 12.1 attempt 0 2024-09-15T02:44:21.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 18 2024-09-15T02:44:21.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 18 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 18 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.651+0000 7f67f8e4b640 20 librados: complete_aio_write 0x5566f1aab3c0 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 20 librados: queue_aio_write 0x5566f195cce0 completion 0x5566f1aab510 write_seq 5 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1aaa3a0 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 19 osd.6 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 19 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 15 client.4423.objecter _send_op 19 to 12.1 on osd.6 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:19 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aaa890 con 0x5566f1a6d330 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.652+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 19 ==== osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) v8 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 19 ondisk uv 6 in 12.1 attempt 0 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 19 2024-09-15T02:44:21.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 19 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 19 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67f8e4b640 20 librados: complete_aio_write 0x5566f1aab510 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 20 librados: queue_aio_write 0x5566f195cce0 completion 0x5566f1aab3c0 write_seq 6 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 10 client.4423.objecter _op_submit op 0x5566f1abd460 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 20 client.4423.objecter _calc_target epoch 22 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 20 client.4423.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 10 client.4423.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [6,3] primary 6 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 20 client.4423.objecter _get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 10 client.4423.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 20 osd.6 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 20 client.4423.objecter get_session s=0x5566f1934df0 osd=6 3 2024-09-15T02:44:21.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 15 client.4423.objecter _session_op_assign 6 20 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 15 client.4423.objecter _send_op 20 to 12.1 on osd.6 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 --> v2:172.21.15.104:6808/33357 -- osd_op(unknown.0.0:20 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) v8 -- 0x5566f1aaad50 con 0x5566f1a6d330 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.653+0000 7f67fc0c1600 5 client.4423.objecter 1 in flight 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 <== osd.6 v2:172.21.15.104:6808/33357 20 ==== osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f67ec01e040 con 0x5566f1a6d330 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 10 client.4423.objecter ms_dispatch 0x5566f193a5b0 osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) v8 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 10 client.4423.objecter in handle_osd_op_reply 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 7 client.4423.objecter handle_osd_op_reply 20 ondisk uv 7 in 12.1 attempt 0 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 10 client.4423.objecter op 0 rval 0 len 0 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 15 client.4423.objecter handle_osd_op_reply completed tid 20 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 15 client.4423.objecter _finish_op 20 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 4 2024-09-15T02:44:21.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 15 client.4423.objecter _session_op_remove 6 20 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 5 client.4423.objecter 0 in flight 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 20 librados: complete_aio_write 0x5566f1aab3c0 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 10 librados: watch_flush enter 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 10 librados: watch_flush exit 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 10 client.4423.objecter close_session for osd.6 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6808/33357 conn(0x5566f1a6d330 msgr2=0x5566f1aa9210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6808/33357 conn(0x5566f1a6d330 0x5566f1aa9210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 20 client.4423.objecter put_session s=0x5566f1934df0 osd=6 1 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 20 client.4423.objecter shutdown clearing up homeless session... 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 10 client.4423.objecter successfully canceled tick 2024-09-15T02:44:21.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6816/32804 conn(0x7f67e406cb30 msgr2=0x7f67e406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/2900149002 >> v2:172.21.15.104:6816/32804 conn(0x7f67e406cb30 0x7f67e406efe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x5566f19a6400 tx=0x7f67f0057000 comp rx=0 tx=0).stop 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 10 monclient: shutdown 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b3710 msgr2=0x5566f1a6dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 --2- 172.21.15.53:0/2900149002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5566f19b3710 0x5566f1a6dda0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f67e800bfd0 tx=0x7f67e8005a80 comp rx=0 tx=0).stop 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67f8e4b640 1 -- 172.21.15.53:0/2900149002 reap_dead start 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 shutdown_connections 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 >> 172.21.15.53:0/2900149002 conn(0x5566f19f2500 msgr2=0x5566f19ed990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.655+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 shutdown_connections 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.656+0000 7f67fc0c1600 1 -- 172.21.15.53:0/2900149002 wait complete. 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.656+0000 7f67fc0c1600 1 librados: shutdown 2024-09-15T02:44:21.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [----------] 12 tests from AsioRados (31 ms total) 2024-09-15T02:44:21.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: 2024-09-15T02:44:21.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [----------] Global test environment tear-down 2024-09-15T02:44:21.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2544 ms total) 2024-09-15T02:44:21.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] Global test environment set-up. 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosList.ListObjects (199 ms) 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (27 ms) 2024-09-15T02:44:21.994 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset foo1,foo2,foo3 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo1 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo2 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo3 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset foo1,foo4,foo5 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo4 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo5 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo1 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset foo6,foo7 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo7 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo6 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-15T02:44:21.995 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns1:foo4 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns1:foo5 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns2:foo7 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns2:foo6 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns1:foo1 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: :foo1 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: :foo2 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: :foo3 2024-09-15T02:44:21.996 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (94 ms) 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 1 0 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 10 0 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 13 0 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 7 0 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 14 0 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 0 0 2024-09-15T02:44:21.997 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 15 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 11 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 5 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 8 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 6 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 3 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 4 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 12 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 9 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 2 0 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (93 ms) 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-15T02:44:21.998 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: x cursor=MIN 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=1 cursor=11:02547ec2:::1:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=10 cursor=11:52ea6a34:::10:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=13 cursor=11:566253c9:::13:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=7 cursor=11:5c6b0b28:::7:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=14 cursor=11:62a1935d:::14:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=0 cursor=11:6cac518f:::0:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=15 cursor=11:863748b0:::15:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=11 cursor=11:89d3ae78:::11:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=5 cursor=11:b29083e3:::5:head 2024-09-15T02:44:21.999 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=8 cursor=11:bd63b0f1:::8:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=6 cursor=11:c4fdafeb:::6:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=3 cursor=11:cfc208b3:::3:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=4 cursor=11:d83876eb:::4:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=12 cursor=11:de5d7c5f:::12:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=9 cursor=11:e960b815:::9:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > oid=2 cursor=11:f905c69b:::2:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=1 cursor=11:02547ec2:::1:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:02547ec2:::1:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:02547ec2:::1:head -> 1 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=10 cursor=11:52ea6a34:::10:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:52ea6a34:::10:head 2024-09-15T02:44:22.000 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2024-09-15T02:44:22.001 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=13 cursor=11:566253c9:::13:head 2024-09-15T02:44:22.001 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:566253c9:::13:head 2024-09-15T02:44:22.001 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:566253c9:::13:head -> 13 2024-09-15T02:44:22.001 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=7 cursor=11:5c6b0b28:::7:head 2024-09-15T02:44:22.001 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2024-09-15T02:44:22.001 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=14 cursor=11:62a1935d:::14:head 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:62a1935d:::14:head 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:62a1935d:::14:head -> 14 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=0 cursor=11:6cac518f:::0:head 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:6cac518f:::0:head 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:6cac518f:::0:head -> 0 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=15 cursor=11:863748b0:::15:head 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:863748b0:::15:head 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:863748b0:::15:head -> 15 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=11 cursor=11:89d3ae78:::11:head 2024-09-15T02:44:22.059 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:89d3ae78:::11:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:89d3ae78:::11:head -> 11 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=5 cursor=11:b29083e3:::5:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:b29083e3:::5:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:b29083e3:::5:head -> 5 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=8 cursor=11:bd63b0f1:::8:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:bd63b0f1:::8:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:bd63b0f1:::8:head -> 8 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=6 cursor=11:c4fdafeb:::6:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:c4fdafeb:::6:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:c4fdafeb:::6:head -> 6 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=3 cursor=11:cfc208b3:::3:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:cfc208b3:::3:head 2024-09-15T02:44:22.060 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:cfc208b3:::3:head -> 3 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=4 cursor=11:d83876eb:::4:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:d83876eb:::4:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:d83876eb:::4:head -> 4 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=12 cursor=11:de5d7c5f:::12:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:de5d7c5f:::12:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:de5d7c5f:::12:head -> 12 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=9 cursor=11:e960b815:::9:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:e960b815:::9:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:e960b815:::9:head -> 9 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : oid=2 cursor=11:f905c69b:::2:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:f905c69b:::2:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:f905c69b:::2:head -> 2 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:566253c9:::13:head 2024-09-15T02:44:22.061 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:566253c9:::13:head expected=11:566253c9:::13:head 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:566253c9:::13:head -> 13 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=13 expected=13 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:5c6b0b28:::7:head expected=11:5c6b0b28:::7:head 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=7 expected=7 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:52ea6a34:::10:head 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:52ea6a34:::10:head expected=11:52ea6a34:::10:head 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=10 expected=10 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:e960b815:::9:head 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:e960b815:::9:head expected=11:e960b815:::9:head 2024-09-15T02:44:22.062 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:e960b815:::9:head -> 9 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=9 expected=9 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:de5d7c5f:::12:head 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:de5d7c5f:::12:head expected=11:de5d7c5f:::12:head 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:de5d7c5f:::12:head -> 12 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=12 expected=12 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:d83876eb:::4:head 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:d83876eb:::4:head expected=11:d83876eb:::4:head 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:d83876eb:::4:head -> 4 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=4 expected=4 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:cfc208b3:::3:head 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:cfc208b3:::3:head expected=11:cfc208b3:::3:head 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:cfc208b3:::3:head -> 3 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=3 expected=3 2024-09-15T02:44:22.063 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:c4fdafeb:::6:head 2024-09-15T02:44:22.064 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:c4fdafeb:::6:head expected=11:c4fdafeb:::6:head 2024-09-15T02:44:22.064 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:c4fdafeb:::6:head -> 6 2024-09-15T02:44:22.338 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : e list_parallel: process_1_[38747]: starting. 2024-09-15T02:44:22.338 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_1_[38747]: creating pool ceph_test_rados_list_parallel.smithi053-38712 2024-09-15T02:44:22.338 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_1_[38747]: created object 0... 2024-09-15T02:44:22.338 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_1_[38747]: created object 25... 2024-09-15T02:44:22.338 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_1_[38747]: created object 49... 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_1_[38747]: finishing. 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_1_[38747]: shutting down. 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_2_[38748]: starting. 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_2_[38748]: listing objects. 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_2_[38748]: listed object 0... 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_2_[38748]: listed object 25... 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_2_[38748]: saw 50 objects 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_2_[38748]: shutting down. 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_3_[39130]: starting. 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_3_[39130]: creating pool ceph_test_rados_list_parallel.smithi053-38712 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_3_[39130]: created object 0... 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_3_[39130]: created object 25... 2024-09-15T02:44:22.339 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_3_[39130]: created object 49... 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_3_[39130]: finishing. 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_3_[39130]: shutting down. 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_4_[39131]: starting. 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_4_[39131]: listing objects. 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_4_[39131]: listed object 0... 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_4_[39131]: listed object 25... 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_4_[39131]: saw 45 objects 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_4_[39131]: shutting down. 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_5_[39132]: starting. 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_5_[39132]: removed 25 objects... 2024-09-15T02:44:22.340 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_5_[39132]: removed half of the objects 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_5_[39132]: removed 50 objects... 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_5_[39132]: removed 50 objects 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_5_[39132]: shutting down. 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_6_[39181]: starting. 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_6_[39181]: creating pool ceph_test_rados_list_parallel.smithi053-38712 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_6_[39181]: created object 0... 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_6_[39181]: created object 25... 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_6_[39181]: created object 49... 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_6_[39181]: finishing. 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_6_[39181]: shutting down. 2024-09-15T02:44:22.341 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_7_[39182]: starting. 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_7_[39182]: listing objects. 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_7_[39182]: listed object 0... 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_7_[39182]: listed object 25... 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_7_[39182]: listed object 50... 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_7_[39182]: saw 55 objects 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_7_[39182]: shutting down. 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_8_[39183]: starting. 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_8_[39183]: added 25 objects... 2024-09-15T02:44:22.342 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_8_[39183]: added half of the objects 2024-09-15T02:44:22.343 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_8_[39183]: added 50 objects... 2024-09-15T02:44:22.343 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_8_[39183]: added 50 objects 2024-09-15T02:44:22.343 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_8_[39183]: shutting down. 2024-09-15T02:44:22.343 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.343 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.490 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ********************** api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (3128 ms) 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (3128 ms total) 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (3128 ms total) 2024-09-15T02:44:22.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (140 ms) 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-15T02:44:22.498 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (26 ms) 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (19 ms) 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (32 ms) 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (12 ms) 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-15T02:44:22.499 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (14 ms) 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (8 ms) 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-09-15T02:44:22.500 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (212 ms) 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (212 ms) 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (6 ms) 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (712 ms total) 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: 2024-09-15T02:44:22.501 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (3130 ms total) 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (431 ms) 2024-09-15T02:44:22.502 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (20 ms) 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2757 ms) 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3209 ms total) 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3209 ms total) 2024-09-15T02:44:22.503 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (57 ms) 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-15T02:44:22.616 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (327 ms) 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2818 ms) 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.433320+0000 mon.a [WRN] Health check failed: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.445597+0000 mon.a [INF] from='client.? 172.21.15.53:0/3987168941' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi053-38012-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.484649+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38673-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.484794+0000 mon.a [INF] from='client.? 172.21.15.53:0/2262063845' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi053-38040-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.484873+0000 mon.a [INF] from='client.? 172.21.15.53:0/194021842' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi053-38318-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.484944+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38438-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.485026+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi053-38275-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.485143+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38215-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.485229+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi053-38679-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.617 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.485316+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38528-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.618 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.485399+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi053-38344-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.618 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.485652+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:22.788 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.485778+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi053-38384-1","app": "rados","yes_i_really_mean_it": true}********* 2024-09-15T02:44:22.788 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_9_[39232]: starting. 2024-09-15T02:44:22.788 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_9_[39232]: creating pool ceph_test_rados_list_parallel.smithi053-38712 2024-09-15T02:44:22.788 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_9_[39232]: created object 0... 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_9_[39232]: created object 25... 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_9_[39232]: created object 49... 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_9_[39232]: finishing. 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_9_[39232]: shutting down. 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: starting. 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: listing objects. 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: listed object 0... 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: listed object 25... 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: listed object 50... 2024-09-15T02:44:22.789 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: listed object 75... 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: listed object 100... 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: saw 103 objects 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_10_[39233]: shutting down. 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_13_[39236]: starting. 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_13_[39236]: removed 25 objects... 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_13_[39236]: removed half of the objects 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_13_[39236]: removed 50 objects... 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_13_[39236]: removed 50 objects 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_13_[39236]: shutting down. 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.790 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_12_[39235]: starting. 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_12_[39235]: added 25 objects... 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_12_[39235]: added half of the objects 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_12_[39235]: added 50 objects... 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_12_[39235]: added 50 objects 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_12_[39235]: shutting down. 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.791 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_11_[39234]: starting. 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_11_[39234]: added 25 objects... 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_11_[39234]: added half of the objects 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_11_[39234]: added 50 objects... 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_11_[39234]: added 50 objects 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_11_[39234]: shutting down. 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_14_[39444]: starting. 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_14_[39444]: creating pool ceph_test_rados_list_parallel.smithi053-38712 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_14_[39444]: created object 0... 2024-09-15T02:44:22.792 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_14_[39444]: created object 25... 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_14_[39444]: created object 49... 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_14_[39444]: finishing. 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_14_[39444]: shutting down. 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: starting. 2024-09-15T02:44:22.793 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: listing objects. 2024-09-15T02:44:23.459 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: listed objec]': finished 2024-09-15T02:44:23.459 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.491029+0000 mon.b [INF] from='client.? 172.21.15.53:0/1743055507' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38165-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:23.459 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.499729+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38165-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.559904+0000 mon.a [INF] from='client.? 172.21.15.53:0/2489300855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi053-37938-22"}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.569525+0000 mon.a [INF] from='client.? 172.21.15.53:0/2489300855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi053-37938-22"}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.579655+0000 mon.a [INF] from='client.? 172.21.15.53:0/2489300855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi053-37938-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.643235+0000 mon.a [INF] from='client.? 172.21.15.53:0/1945422130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi053-37922-16"}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.647317+0000 mon.a [INF] from='client.? 172.21.15.53:0/1945422130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi053-37922-16"}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:21.688591+0000 mon.a [INF] from='client.? 172.21.15.53:0/1945422130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi053-37922-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.454467+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi053-38429-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi053-38429-1"}]': finished 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.454648+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38165-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.454796+0000 mon.a [INF] from='client.? 172.21.15.53:0/2489300855' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi053-37938-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.454909+0000 mon.a [INF] from='client.? 172.21.15.53:0/1945422130' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi053-37922-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.458013+0000 mon.a [INF] from='client.? 172.21.15.53:0/2489300855' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi053-37938-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi053-37938-22"}]: dispatch 2024-09-15T02:44:23.460 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.458544+0000 mon.a [INF] from='client.? 172.21.15.53:0/1945422130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi053-37922-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi053-37922-16"}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.467989+0000 mon.c [INF] from='client.? 172.21.15.53:0/1753963356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi053-37884-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.512937+0000 mon.a [INF] from='client.? 172.21.15.53:0/1203650533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi053-37912-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.513325+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi053-37884-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.585294+0000 mon.c [INF] from='client.? 172.21.15.53:0/1663285244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi053-38344-6"}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.608314+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi053-38344-6"}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.609838+0000 mon.c [INF] from='client.? 172.21.15.53:0/1663285244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi053-38344-6"}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.613014+0000 mon.c [INF] from='client.? 172.21.15.53:0/3571883514' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.613465+0000 client.admin [INF] onexx 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.613867+0000 mon.c [INF] from='client.? 172.21.15.53:0/1264371156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi053-38318-7"}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.681495+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi053-38344-6"}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.682174+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-15T02:44:23.461 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.682303+0000 mon.c [INF] from='client.? 172.21.15.53:0/1663285244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi053-38344-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:23.462 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.683749+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi053-38318-7"}]: dispatch 2024-09-15T02:44:23.462 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.684104+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi053-38344-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:23.462 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.684484+0000 mon.c [INF] from='client.? 172.21.15.53:0/1264371156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi053-38318-7"}]: dispatch 2024-09-15T02:44:23.462 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.686108+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi053-38318-7"}]: dispatch 2024-09-15T02:44:23.462 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.686725+0000 mon.c [INF] from='client.? 172.21.15.53:0/1264371156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi053-38318-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:23.543 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:22.687367+0000 mon.a [INF] from=t 0... 2024-09-15T02:44:23.543 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: listed object 25... 2024-09-15T02:44:23.543 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: listed object 50... 2024-09-15T02:44:23.543 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: listed object 75... 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: listed object 100... 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: listed object 125... 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: saw 150 objects 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_15_[39445]: shutting down. 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_16_[39446]: starting. 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_16_[39446]: added 25 objects... 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_16_[39446]: added half of the objects 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_16_[39446]: added 50 objects... 2024-09-15T02:44:23.544 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_16_[39446]: added 50 objects 2024-09-15T02:44:23.545 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: process_16_[39446]: shutting down. 2024-09-15T02:44:23.545 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.545 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.545 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.545 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.545 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******************************* 2024-09-15T02:44:23.545 INFO:tasks.workunit.client.0.smithi053.stdout: list_parallel: ******* SUCCESS ********** 2024-09-15T02:44:25.524 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_1_[38775]: starting. 2024-09-15T02:44:25.524 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_1_[38775]: creating pool ceph_test_rados_open_pools_parallel.smithi053-38741 2024-09-15T02:44:25.524 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_1_[38775]: created object 0... 2024-09-15T02:44:25.524 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_1_[38775]: created object 25... 2024-09-15T02:44:25.524 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_1_[38775]: created object 49... 2024-09-15T02:44:25.524 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_1_[38775]: finishing. 2024-09-15T02:44:25.524 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_1_[38775]: shutting down. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_2_[38776]: starting. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_2_[38776]: rados_pool_create. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_2_[38776]: rados_ioctx_create. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_2_[38776]: shutting down. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: ******************************* 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_3_[39318]: starting. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_3_[39318]: creating pool ceph_test_rados_open_pools_parallel.smithi053-38741 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_3_[39318]: created object 0... 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_3_[39318]: created object 25... 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_3_[39318]: created object 49... 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_3_[39318]: finishing. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_3_[39318]: shutting down. 2024-09-15T02:44:25.525 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: ******************************* 2024-09-15T02:44:25.526 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_4_[39319]: starting. 2024-09-15T02:44:25.526 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_4_[39319]: rados_pool_create. 2024-09-15T02:44:25.526 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_4_[39319]: rados_ioctx_create. 2024-09-15T02:44:25.526 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: process_4_[39319]: shutting down. 2024-09-15T02:44:25.526 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: ******************************* 2024-09-15T02:44:25.526 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: ******************************* 2024-09-15T02:44:25.526 INFO:tasks.workunit.client.0.smithi053.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-15T02:44:25.531 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_1_[38812]: starting. 2024-09-15T02:44:25.531 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_1_[38812]: creating pool ceph_test_rados_delete_pools_parallel.smithi053-38787 2024-09-15T02:44:25.531 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_1_[38812]: created object 0... 2024-09-15T02:44:25.531 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_1_[38812]: created object 25... 2024-09-15T02:44:25.531 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_1_[38812]: created object 49... 2024-09-15T02:44:25.531 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_1_[38812]: finishing. 2024-09-15T02:44:25.531 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_1_[38812]: shutting down. 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_2_[38815]: starting. 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_2_[38815]: deleting pool ceph_test_rados_delete_pools_parallel.smithi053-38787 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_2_[38815]: shutting down. 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: ******************************* 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_3_[39320]: starting. 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_3_[39320]: creating pool ceph_test_rados_delete_pools_parallel.smithi053-38787 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_3_[39320]: created object 0... 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_3_[39320]: created object 25... 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_3_[39320]: created object 49... 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_3_[39320]: finishing. 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_3_[39320]: shutting down. 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: ******************************* 2024-09-15T02:44:25.532 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_5_[39322]: starting. 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_5_[39322]: listing objects. 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_5_[39322]: listed object 0... 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_5_[39322]: listed object 25... 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_5_[39322]: saw 50 objects 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_5_[39322]: shutting down. 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: ******************************* 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_4_[39321]: starting. 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_4_[39321]: deleting pool ceph_test_rados_delete_pools_parallel.smithi053-38787 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: process_4_[39321]: shutting down. 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: ******************************* 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: ******************************* 2024-09-15T02:44:25.533 INFO:tasks.workunit.client.0.smithi053.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-15T02:44:25.586 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: seed 37938 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (106 ms) 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (20 ms) 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-15T02:44:25.587 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (5 ms) 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (2 ms) 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2 ms) 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (3 ms) 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-09-15T02:44:25.588 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4 ms) 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3 ms) 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (4 ms) 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (4 ms) 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (9 ms) 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2024-09-15T02:44:25.589 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (3 ms) 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (186 ms total) 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (977 ms) 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-15T02:44:25.590 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (21 ms) 2024-09-15T02:44:25.591 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-15T02:44:25.591 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-09-15T02:44:25.591 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-15T02:44:25.591 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-09-15T02:44:25.591 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-15T02:44:25.591 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (2 ms) 2024-09-15T02:44:26.576 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] L'client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi053-38318-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.351737+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.0"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.352311+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.1"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.352753+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.2"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.353126+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.3"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.353752+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.4"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.354149+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.5"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.354531+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.6"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.354897+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.7"}]: dispatch 2024-09-15T02:44:26.577 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.355314+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.8"}]: dispatch 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:23.355685+0000 mon.b [INF] from='client.? 172.21.15.53:0/4260418549' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "9.9"}]: dispatch 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.464149+0000 mon.a [INF] from='client.? 172.21.15.53:0/2489300855' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi053-37938-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi053-37938-22"}]': finished 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.464234+0000 mon.a [INF] from='client.? 172.21.15.53:0/1945422130' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi053-37922-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi053-37922-16"}]': finished 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.464395+0000 mon.a [INF] from='client.? 172.21.15.53:0/3130758483' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.464517+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38215-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.464584+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38165-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.468406+0000 mon.b [INF] from='client.? 172.21.15.53:0/1743055507' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi053-38165-4", "tierpool": "test-rados-api-smithi053-38165-4-cache"}]: dispatch 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.471211+0000 mon.c [INF] from='client.? 172.21.15.53:0/669738968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi053-38429-1"}]: dispatch 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.561090+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi053-37985-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.561980+0000 mon.b [INF] from='client.? 172.21.15.53:0/1615671009' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi053-37985-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi053-37985-10"}]: dispatch 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.564580+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi053-38165-4", "tierpool": "test-rados-api-smithi053-38165-4-cache"}]: dispatch 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.564919+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi053-38429-1"}]: dispatch 2024-09-15T02:44:26.578 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.574349+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.575609+0000 mon.a [INF] from='client.? 172.21.15.53:0/1203650533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi053-37912-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.627534+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi053-37985-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi053-37985-10"}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.630649+0000 mon.b [INF] from='client.? 172.21.15.53:0/895750043' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi053-38012-10"}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.666808+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi053-38012-10"}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.667503+0000 mon.b [INF] from='client.? 172.21.15.53:0/895750043' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi053-38012-10"}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.667941+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi053-38012-10"}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.668920+0000 mon.b [INF] from='client.? 172.21.15.53:0/895750043' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi053-38012-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:24.669485+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi053-38012-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T02:44:26.579 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.467519+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi053-38344-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi053-38344-6"}]': finished 2024-09-15T02:44:26.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.467644+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi053-38318-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi053-38318-7"}]': finished 2024-09-15T02:44:26.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.467748+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi053-38165-4", "tierpool": "test-rados-api-smithi053-38165-4-cache"}]': finished 2024-09-15T02:44:26.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.467843+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi053-38429-1"}]': finished 2024-09-15T02:44:26.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.467975+0000 mon.a [INF] from='client.? 172.21.15.53:0/1203650533' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi053-37912-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:26.636 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.472764+0000 mon.c [INF] from='client.? 172.21.15.53:0/669738968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi053-38429-1"}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.477623+0000 mon.b [INF] from='client.? 172.21.15.53:0/1743055507' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi053-38165-4", "overlaypool": "test-rados-api-smithi053-38165-4-cache"}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.510626+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi053-38012-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.511563+0000 mon.b [INF] from='client.? 172.21.15.53:0/895750043' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi053-38012-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi053-38012-10"}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.513452+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi053-38165-4", "overlaypool": "test-rados-api-smithi053-38165-4-cache"}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.513781+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi053-38429-1"}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.516666+0000 mon.b [INF] from='client.? 172.21.15.53:0/4220431469' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi053-37884-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.522545+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi053-38012-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi053-38012-10"}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.529351+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi053-37884-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:25.537420+0000 mon.a [INF] from='client.? 172.21.15.53:0/1203650533' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi053-37912-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-15T02:44:26.637 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.470692+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi053-37985-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi053-37985-10"}]': finished 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (12 ms) 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (12 ms) 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (6 ms) 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-15T02:44:28.491 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (9 ms) 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (9 ms) 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (8 ms) 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (11 ms) 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (16 ms) 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (4 ms) 2024-09-15T02:44:28.492 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1106 ms total) 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (9420 ms total) 2024-09-15T02:44:28.493 INFO:tasks.workunit.client.0.smithi053.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-15T02:44:28.508 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: Running main() from gmock_main.cc 2024-09-15T02:44:28.508 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-15T02:44:28.508 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [----------] Global test environment set-up. 2024-09-15T02:44:28.508 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-15T02:44:28.508 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-15T02:44:28.508 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (109 ms) 2024-09-15T02:44:28.508 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: no timeout :/ 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: no timeout :/ 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: no timeout :/ 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: no timeout :/ 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: no timeout :/ 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (36 ms) 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5 ms) 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-09-15T02:44:28.509 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3 ms) 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (3 ms) 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.TruncTest (3 ms) 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2024-09-15T02:44:28.510 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.RmXattr (11 ms) 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIo.XattrIter (5 ms) 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [----------] 14 tests from LibRadosIo (192 ms total) 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (984 ms) 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (6 ms) 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (5 ms) 2024-09-15T02:44:28.511 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (11 ms) 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (14 ms) 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (9 ms) 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (13 ms) 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (16 ms) 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-15T02:44:28.512 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (15 ms) 2024-09-15T02:44:28.513 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1079 ms total) 2024-09-15T02:44:28.513 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: 2024-09-15T02:44:28.516 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [----------] Global test environment tear-down 2024-09-15T02:44:28.516 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9438 ms total) 2024-09-15T02:44:28.517 INFO:tasks.workunit.client.0.smithi053.stdout: api_io: [ PASSED ] 24 tests. 2024-09-15T02:44:29.338 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-15T02:44:29.338 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-15T02:44:29.338 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-15T02:44:29.338 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-15T02:44:29.338 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: seed 38318 2024-09-15T02:44:29.338 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (153 ms) 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (8 ms) 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (18 ms) 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (7 ms) 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (14 ms) 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (201 ms total) 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-15T02:44:29.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1061 ms) 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (5 ms) 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (13 ms) 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1080 ms total) 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10115 ms total) 2024-09-15T02:44:29.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-15T02:44:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: Running main() from gmock_main.cc 2024-09-15T02:44:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-09-15T02:44:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [----------] Global test environment set-up. 2024-09-15T02:44:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-09-15T02:44:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-15T02:44:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStat.Stat (116 ms) 2024-09-15T02:44:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-15T02:44:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStat.StatNS (54 ms) 2024-09-15T02:44:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-15T02:44:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-15T02:44:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-15T02:44:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (6 ms) 2024-09-15T02:44:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [----------] 4 tests from LibRadosStat (177 ms total) 2024-09-15T02:44:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: 2024-09-15T02:44:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-15T02:44:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-15T02:44:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1042 ms) 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (24 ms) 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3 ms) 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1070 ms total) 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [----------] Global test environment tear-down 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (10123 ms total) 2024-09-15T02:44:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_stat: [ PASSED ] 8 tests. 2024-09-15T02:44:29.643 INFO:tasks.workunit.client.0.smithi053.stdout:d: got: 2024-09-15T02:44:26.470908+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi053-38165-4", "overlaypool": "test-rados-api-smithi053-38165-4-cache"}]': finished 2024-09-15T02:44:29.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.471023+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi053-38429-1"}]': finished 2024-09-15T02:44:29.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.471747+0000 mon.b [INF] from='client.? 172.21.15.53:0/1743055507' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi053-38165-4-cache", "mode": "writeback"}]: dispatch 2024-09-15T02:44:29.643 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.515274+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi053-37884-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.515452+0000 mon.a [INF] from='client.? 172.21.15.53:0/1203650533' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi053-37912-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.519005+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi053-38165-4-cache", "mode": "writeback"}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.525903+0000 mon.c [INF] from='client.? 172.21.15.53:0/1595706380' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.526258+0000 mon.a [INF] from='client.? 172.21.15.53:0/1945422130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi053-37922-16"}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.526967+0000 mon.c [INF] from='client.? 172.21.15.53:0/2447350159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38215-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.534604+0000 mon.a [INF] from='client.? 172.21.15.53:0/2489300855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi053-37938-22"}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.535535+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.535880+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi053-38215-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.575970+0000 mon.c [INF] from='client.? 172.21.15.53:0/3571883514' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.576101+0000 client.admin [INF] threexx 2024-09-15T02:44:29.644 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: got: 2024-09-15T02:44:26.580096+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-15T02:44:29.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7172 ms) 2024-09-15T02:44:29.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10375 ms total) 2024-09-15T02:44:29.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: 2024-09-15T02:44:29.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-15T02:44:29.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10375 ms total) 2024-09-15T02:44:29.645 INFO:tasks.workunit.client.0.smithi053.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: Running main() from gmock_main.cc 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [----------] Global test environment set-up. 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (43 ms) 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.LockShared (55 ms) 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1101 ms) 2024-09-15T02:44:32.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1029 ms) 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5 ms) 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (9 ms) 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [----------] 8 tests from LibRadosLock (2249 ms total) 2024-09-15T02:44:32.353 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1061 ms) 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (3 ms) 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1009 ms) 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1006 ms) 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (7 ms) 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-09-15T02:44:32.354 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (10 ms) 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3105 ms total) 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [----------] Global test environment tear-down 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13260 ms total) 2024-09-15T02:44:32.355 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: seed 38012 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (148 ms) 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-15T02:44:33.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (7 ms) 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1029 ms) 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1004 ms) 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (7 ms) 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (8 ms) 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (8 ms) 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (2 ms) 2024-09-15T02:44:33.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2213 ms total) 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1062 ms) 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (8 ms) 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1008 ms) 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1006 ms) 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-15T02:44:33.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3100 ms total) 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (14267 ms total) 2024-09-15T02:44:33.382 INFO:tasks.workunit.client.0.smithi053.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: notify 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1056 ms) 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (10 ms) 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1066 ms total) 2024-09-15T02:44:35.437 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: notify 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (68 ms) 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: notify 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3833 ms) 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (6 ms) 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (6 ms) 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675728989744 notify_id 141733920768 notifier_gid 4751 2024-09-15T02:44:35.438 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (9 ms) 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675729774256 notify_id 141733920769 notifier_gid 4751 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (6 ms) 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675729769504 notify_id 141733920770 notifier_gid 4751 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5 ms) 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675729769440 notify_id 141733920770 notifier_gid 4751 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (7 ms) 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675740825632 notify_id 141733920771 notifier_gid 4751 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (5 ms) 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-15T02:44:35.439 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675740825632 notify_id 141733920770 notifier_gid 4751 2024-09-15T02:44:35.440 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-09-15T02:44:35.440 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-15T02:44:35.440 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: trying... 2024-09-15T02:44:35.440 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675729769440 notify_id 141733920772 notifier_gid 4751 2024-09-15T02:44:35.440 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: timed out 2024-09-15T02:44:35.440 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: flushing 2024-09-15T02:44:38.358 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notif api_watch_notify: Running main() from gmock_main.cc 2024-09-15T02:44:38.358 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-15T02:44:38.358 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-15T02:44:38.358 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-15T02:44:38.358 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify_test_cb 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (131 ms) 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94133887255440 err -107 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (25 ms) 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94133887255440 err -107 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-15T02:44:38.359 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (16 ms) 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 94489280512 cookie 94133887267008 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (12 ms) 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 94489280512 cookie 94133887324192 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (17 ms) 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 94489280513 cookie 94133887324192 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (12 ms) 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 94489280512 cookie 94133887302912 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 94489280512 cookie 94133887337200 2024-09-15T02:44:38.360 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (22 ms) 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 94489280512 cookie 94133887302912 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 98784247809 cookie 94133887302912 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3013 ms) 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94133887337120 err -107 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_cb from 4459 notify_id 128849018883 cookie 94133887337120 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94133887302912 err -107 2024-09-15T02:44:38.361 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9262 ms total) 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: watch_notify_test_cb 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1050 ms) 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1050 ms total) 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (19112 ms total) 2024-09-15T02:44:38.362 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-15T02:44:40.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3151 ms) 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3058 ms) 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2962 ms) 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2825 ms) 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3005 ms) 2024-09-15T02:44:40.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-15T02:44:40.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3034 ms) 2024-09-15T02:44:40.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-15T02:44:40.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (2999 ms) 2024-09-15T02:44:40.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (21035 ms total) 2024-09-15T02:44:40.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: 2024-09-15T02:44:40.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-15T02:44:40.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (21035 ms total) 2024-09-15T02:44:40.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-15T02:44:46.822 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-15T02:44:46.822 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [----------] Global test environment set-up. 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5128 ms) 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ OK ] LibRadosService.RegisterLate (16 ms) 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: cluster: 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: id: 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: health: HEALTH_WARN 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 5 pool(s) do not have an application enabled 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: services: 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 75s) 2024-09-15T02:44:46.823 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: mgr: x(active, since 71s), standbys: y 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: osd: 8 osds: 8 up (since 66s), 8 in (since 73s) 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: data: 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: pools: 16 pools, 433 pgs 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: usage: 2.3 GiB used, 718 GiB / 720 GiB avail 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: pgs: 97.921% pgs unknown 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 424 unknown 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 9 active+clean 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.824 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: cluster: 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: id: 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: health: HEALTH_WARN 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 7 pool(s) do not have an application enabled 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: services: 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 77s) 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: mgr: x(active, since 73s), standbys: y 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: osd: 8 osds: 8 up (since 68s), 8 in (since 75s) 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-15T02:44:46.825 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: data: 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: pools: 20 pools, 537 pgs 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: objects: 61 objects, 145 MiB 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: usage: 2.3 GiB used, 718 GiB / 720 GiB avail 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: pgs: 18.808% pgs unknown 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 26.816% pgs not active 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 292 active+clean 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 144 creating+peering 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 101 unknown 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: io: 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: client: 12 MiB/s rd, 37 MiB/s wr, 254 op/s rd, 291 op/s wr 2024-09-15T02:44:46.826 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2325 ms) 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ OK ] LibRadosService.Status (20032 ms) 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [----------] 4 tests from LibRadosService (27501 ms total) 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [----------] Global test environment tear-down 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27501 ms total) 2024-09-15T02:44:46.827 INFO:tasks.workunit.client.0.smithi053.stdout: api_service: [ PASSED ] 4 tests. 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout:y_pp: flushed 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: trying... 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675740825632 notify_id 154618822659 notifier_gid 4751 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: timed out 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: flushing 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: flushed 2024-09-15T02:44:48.367 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3206 ms) 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: List watches 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: notify2 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675729769440 notify_id 167503724548 notifier_gid 4751 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: notify2 done 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: watch_check 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: unwatch2 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: flushing 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: done 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3009 ms) 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: List watches 2024-09-15T02:44:48.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: notify2 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: handle_notify cookie 94675729769440 notify_id 180388626434 notifier_gid 4751 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: notify2 done 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: watch_check 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: unwatch2 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: flushing 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: done 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3013 ms) 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16182 ms total) 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (29106 ms total) 2024-09-15T02:44:48.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5047 ms) 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (28 ms) 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20026 ms) 2024-09-15T02:44:54.957 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: attempt 0 of 20 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: failed to deregister: 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: { 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "epoch": 8, 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "modified": "2024-09-15T02:44:47.349328+0000", 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "services": { 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "laundry": { 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "daemons": { 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "summary": "", 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "close-test-pid38605": { 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "start_epoch": 7, 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "start_stamp": "2024-09-15T02:44:44.522095+0000", 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "gid": 12501, 2024-09-15T02:44:54.958 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "addr": "172.21.15.53:0/830312671", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "metadata": { 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "arch": "x86_64", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "ceph_release": "quincy", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "distro": "centos", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "distro_version": "9", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "foo": "bar", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "hostname": "smithi053", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T02:44:54.959 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "mem_swap_kb": "0", 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "mem_total_kb": "32493492", 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "os": "Linux", 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "this": "that" 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: }, 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: "task_status": {} 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: } 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: } 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: } 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: } 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: } 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: 2024-09-15T02:44:54.960 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: attempt 1 of 20 2024-09-15T02:44:54.961 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10518 ms) 2024-09-15T02:44:54.961 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (35619 ms total) 2024-09-15T02:44:54.961 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: 2024-09-15T02:44:54.961 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-15T02:44:54.961 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (35619 ms total) 2024-09-15T02:44:54.961 INFO:tasks.workunit.client.0.smithi053.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: Running main() from gmock_main.cc 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [----------] Global test environment set-up. 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3298 ms) 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3026 ms) 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-15T02:44:57.384 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2996 ms) 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2846 ms) 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2992 ms) 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5032 ms) 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4975 ms) 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5015 ms) 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8028 ms) 2024-09-15T02:44:57.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [----------] 9 tests from LibRadosPools (38208 ms total) 2024-09-15T02:44:57.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: 2024-09-15T02:44:57.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [----------] Global test environment tear-down 2024-09-15T02:44:57.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (38208 ms total) 2024-09-15T02:44:57.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-15T02:44:58.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-15T02:44:58.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] Global test environment set-up. 2024-09-15T02:44:58.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: 2024-09-15T02:44:19.202+0000 7ff53fa33600 -1 failed for service _ceph-mon._tcp 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: 2024-09-15T02:44:19.202+0000 7ff53fa33600 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-15T02:44:58.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (74 ms) 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5020 ms) 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5094 ms total) 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: started 0x558de3de2ad0 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: started 0x558de3e736d0 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: started 2 aios 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: waiting 0x558de3de2ad0 2024-09-15T02:44:58.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: waiting 0x558de3e736d0 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: done. 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5105 ms) 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5105 ms total) 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.Exec (63 ms) 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-15T02:44:58.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (18 ms) 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (203 ms) 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4730 ms) 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21584 ms) 2024-09-15T02:44:58.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] 8 tests from LibRadosMisc (26598 ms total) 2024-09-15T02:44:58.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: 2024-09-15T02:44:58.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [----------] Global test environment tear-down 2024-09-15T02:44:58.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (39254 ms total) 2024-09-15T02:44:58.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2051 ms) 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-15T02:45:10.339 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (1982 ms) 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1958 ms) 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1835 ms) 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (7828 ms total) 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4018 ms) 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4038 ms) 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8056 ms total) 2024-09-15T02:45:10.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3008 ms) 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1994 ms) 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1999 ms) 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2039 ms) 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9040 ms total) 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-15T02:45:10.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4029 ms) 2024-09-15T02:45:10.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-15T02:45:10.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4004 ms) 2024-09-15T02:45:10.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8033 ms total) 2024-09-15T02:45:10.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: 2024-09-15T02:45:10.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-15T02:45:10.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (51145 ms total) 2024-09-15T02:45:10.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-09-15T02:45:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-15T02:45:29.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: seed 38067 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-15T02:45:29.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (14 ms) 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (109 ms) 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (10 ms) 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (4 ms) 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (7 ms) 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-09-15T02:45:29.348 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (3 ms) 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (9 ms) 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-09-15T02:45:29.349 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4 ms) 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (7 ms) 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2310 ms) 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (210 ms) 2024-09-15T02:45:29.350 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: done waiting, doing copies 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: done waiting 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60448 ms) 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (8 ms) 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (5 ms) 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-15T02:45:29.351 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4761 ms) 2024-09-15T02:45:29.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-15T02:45:29.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-15T02:45:29.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-15T02:45:29.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-15T02:45:29.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-15T02:45:29.352 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (1 ms) 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP ( api_snapshots_pp: Running main() from gmock_main.cc 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: seed 38291 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2005 ms) 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2104 ms) 2024-09-15T02:45:34.368 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1959 ms) 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1959 ms) 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3841 ms) 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11868 ms total) 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4051 ms) 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-15T02:45:34.369 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3991 ms) 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5955 ms) 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4018 ms) 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2014 ms) 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi053-38291-7 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18003 ms) 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (38032 ms total) 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-15T02:45:34.370 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (11 ms) 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5022 ms) 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5033 ms total) 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3000 ms) 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2012 ms) 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1970 ms) 2024-09-15T02:45:34.371 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-15T02:45:34.372 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2013 ms) 2024-09-15T02:45:34.372 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8996 ms total) 2024-09-15T02:45:34.372 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: 2024-09-15T02:45:40.340 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_p api_aio: Running main() from gmock_main.cc 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [----------] Global test environment set-up. 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2451 ms) 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3079 ms) 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2916 ms) 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2817 ms) 2024-09-15T02:45:40.341 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3055 ms) 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2959 ms) 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3032 ms) 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3020 ms) 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2959 ms) 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3002 ms) 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-15T02:45:40.342 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3028 ms) 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2948 ms) 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3061 ms) 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2952 ms) 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.Flush (3010 ms) 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2991 ms) 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2991 ms) 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-15T02:45:40.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3011 ms) 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3024 ms) 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3003 ms) 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3011 ms) 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2981 ms) 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3011 ms) 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2979 ms) 2024-09-15T02:45:40.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-15T02:45:40.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3014 ms) 2024-09-15T02:45:40.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [----------] 25 tests from LibRadosAio (74310 ms total) 2024-09-15T02:45:40.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: 2024-09-15T02:45:40.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-15T02:45:40.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-15T02:45:40.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6978 ms) 2024-09-15T02:45:52.356 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplep: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4000 ms) 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3984 ms) 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3983 ms) 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11967 ms total) 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: 2024-09-15T02:45:52.357 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-15T02:45:52.358 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (93157 ms total) 2024-09-15T02:45:52.358 INFO:tasks.workunit.client.0.smithi053.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout:67920 ms total) 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (89 ms) 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (89 ms total) 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-15T02:45:55.379 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (25 ms) 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (12 ms) 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (37 ms total) 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (37 ms) 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (3 ms) 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (41 ms total) 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-15T02:45:55.380 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (36 ms) 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (5 ms) 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (41 ms total) 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1036 ms) 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1036 ms total) 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (96231 ms total) 2024-09-15T02:45:55.381 INFO:tasks.workunit.client.0.smithi053.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-15T02:45:59.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-15T02:45:59.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-15T02:45:59.343 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2445 ms) 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19862 ms) 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5996 ms) 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2969 ms) 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3035 ms) 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-15T02:45:59.344 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2995 ms) 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2968 ms) 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3009 ms) 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2993 ms) 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3010 ms) 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2990 ms) 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3021 ms) 2024-09-15T02:45:59.345 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3004 ms) 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2989 ms) 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3000 ms) 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3007 ms) 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2985 ms) 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3015 ms) 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3038 ms) 2024-09-15T02:45:59.346 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2982 ms) 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2970 ms) 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3009 ms) 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3021 ms) 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3005 ms) 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2972 ms) 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3048 ms) 2024-09-15T02:45:59.347 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-15T02:46:52.803 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAiontry=6 expected=6 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:bd63b0f1:::8:head 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:bd63b0f1:::8:head expected=11:bd63b0f1:::8:head 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:bd63b0f1:::8:head -> 8 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=8 expected=8 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:b29083e3:::5:head 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:b29083e3:::5:head expected=11:b29083e3:::5:head 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:b29083e3:::5:head -> 5 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=5 expected=5 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:89d3ae78:::11:head 2024-09-15T02:46:52.804 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:89d3ae78:::11:head expected=11:89d3ae78:::11:head 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:89d3ae78:::11:head -> 11 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=11 expected=11 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:863748b0:::15:head 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:863748b0:::15:head expected=11:863748b0:::15:head 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:863748b0:::15:head -> 15 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=15 expected=15 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:6cac518f:::0:head 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:6cac518f:::0:head expected=11:6cac518f:::0:head 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:6cac518f:::0:head -> 0 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=0 expected=0 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:62a1935d:::14:head 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:62a1935d:::14:head expected=11:62a1935d:::14:head 2024-09-15T02:46:52.805 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:62a1935d:::14:head -> 14 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=14 expected=14 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:02547ec2:::1:head 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:02547ec2:::1:head expected=11:02547ec2:::1:head 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:02547ec2:::1:head -> 1 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=1 expected=1 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : seek to 11:f905c69b:::2:head 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : cursor()=11:f905c69b:::2:head expected=11:f905c69b:::2:head 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: > 11:f905c69b:::2:head -> 2 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: : entry=2 expected=2 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (210 ms) 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (115319 ms) 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-15T02:46:52.806 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: split 0/5 -> MIN 11:33333333::::head 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: split 1/5 -> 11:33333333::::head 11:66666666::::head 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: split 2/5 -> 11:66666666::::head 11:99999999::::head 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: split 3/5 -> 11:99999999::::head 11:cccccccc::::head 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: split 4/5 -> 11:cccccccc::::head MAX 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (30005 ms) 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] 7 tests from LibRadosList (145947 ms total) 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1387 ms) 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset foo1,foo2,foo3 2024-09-15T02:46:52.807 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo1 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo2 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo3 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset foo1,foo4,foo5 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo4 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo5 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo1 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset foo6,foo7 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo7 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: foo6 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns1:foo4 2024-09-15T02:46:52.808 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns1:foo5 2024-09-15T02:46:52.809 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns2:foo7 2024-09-15T02:46:52.809 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns2:foo6 2024-09-15T02:46:52.809 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: ns1:foo1 2024-09-15T02:46:52.810 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: :foo1 2024-09-15T02:46:52.810 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: :foo2 2024-09-15T02:46:52.810 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: :foo3 2024-09-15T02:46:52.810 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (57 ms) 2024-09-15T02:46:58.385 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 1 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 10 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 13 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 7 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 14 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 0 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 15 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 11 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 5 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 8 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 6 0 2024-09-15T02:46:58.386 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 3 0 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 4 0 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 12 0 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 9 0 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 2 0 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (107 ms) 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] 3 tests from LibRadosListEC (1551 ms total) 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3018 ms) 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] 1 test from LibRadosListNP (3018 ms total) 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: 2024-09-15T02:46:58.387 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [----------] Global test environment tear-down 2024-09-15T02:46:58.388 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [==========] 11 tests from 3 test suites ran. (159253 ms total) 2024-09-15T02:46:58.388 INFO:tasks.workunit.client.0.smithi053.stdout: api_list: [ PASSED ] 11 tests. 2024-09-15T02:47:25.402 INFO:tasks.workunit.client.0.smithi053.stdout:te 2024-09-15T02:47:25.402 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7022 ms) 2024-09-15T02:47:25.402 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-15T02:47:25.402 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6997 ms) 2024-09-15T02:47:25.402 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-15T02:47:25.402 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6993 ms) 2024-09-15T02:47:25.402 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7018 ms) 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6991 ms) 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6998 ms) 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7001 ms) 2024-09-15T02:47:25.403 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7000 ms) 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7018 ms) 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7019 ms) 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6967 ms) 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6997 ms) 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7012 ms) 2024-09-15T02:47:25.404 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7056 ms) 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6970 ms) 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112037 ms total) 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [----------] Global test environment tear-down 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (186347 ms total) 2024-09-15T02:47:25.405 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio: [ PASSED ] 41 tests. 2024-09-15T02:47:46.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-15T02:47:46.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: seed 38165 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (130 ms) 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10737 ms) 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (10 ms) 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10877 ms total) 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-15T02:47:46.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7024 ms) 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7971 ms) 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10002 ms) 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: my_snaps [4] 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: my_snaps [5,4] 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: my_snaps [6,5,4] 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: promoting some heads 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: promoting from clones for snap 7 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-15T02:47:46.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: waiting for scrubs... 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: done waiting 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46028 ms) 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10004 ms) 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8031 ms) 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7960 ms) 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8000 ms) 2024-09-15T02:47:46.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10007 ms) 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9041 ms) 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9986 ms) 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12980 ms) 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7996 ms) 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8033 ms) 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-15T02:47:46.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12976 ms) 2024-09-15T02:47:46.398 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-15T02:47:46.398 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8010 ms) 2024-09-15T02:47:46.398 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-15T02:47:46.398 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8015 ms) 2024-09-15T02:48:01.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-15T02:48:01.663 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: ok, hit_set contains 188:602f83fe:::foo:head 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9008 ms) 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg_num = 32 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 0 ls 1726368482,0 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 1 ls 1726368482,0 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 2 ls 1726368482,0 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 3 ls 1726368482,0 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 4 ls 1726368482,0 2024-09-15T02:48:01.664 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 5 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 6 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 7 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 8 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 9 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 10 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 11 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 12 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 13 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 14 ls 1726368482,0 2024-09-15T02:48:01.665 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 15 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 16 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 17 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 18 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 19 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 20 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 21 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 22 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 23 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 24 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 25 ls 1726368482,0 2024-09-15T02:48:01.666 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 26 ls 1726368482,0 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 27 ls 1726368482,0 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 28 ls 1726368482,0 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 29 ls 1726368482,0 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 30 ls 1726368482,0 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg 31 ls 1726368482,0 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: pg_num = 32 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6cac518f:::0:head 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:02547ec2:::1:head 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f905c69b:::2:head 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:cfc208b3:::3:head 2024-09-15T02:48:01.667 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d83876eb:::4:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b29083e3:::5:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c4fdafeb:::6:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:5c6b0b28:::7:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:bd63b0f1:::8:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:e960b815:::9:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:52ea6a34:::10:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:89d3ae78:::11:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:de5d7c5f:::12:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:566253c9:::13:head 2024-09-15T02:48:01.668 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:62a1935d:::14:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:863748b0:::15:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:3958e169:::16:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:4d4dabf9:::17:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:8391935d:::18:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:28883081:::19:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:69259c59:::20:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:4bdb80b7:::21:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a11c5d71:::22:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:271af37b:::23:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:95b121be:::24:head 2024-09-15T02:48:01.669 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:58d1031b:::25:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:0a050783:::26:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c709704c:::27:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:cbe56eaf:::28:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:86b4b162:::29:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:70d89383:::30:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:dd450c7c:::31:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6d5729b1:::32:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c388f3fb:::33:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:56cfea31:::34:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:9dbc1bf7:::35:head 2024-09-15T02:48:01.670 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:40b74ccd:::36:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:4d5aaf42:::37:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:920f362c:::38:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6cc53222:::39:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:9cad833f:::40:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:1ea84d41:::41:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c4480ef6:::42:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a694361e:::43:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d1bd33e9:::44:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:ddc2cd5d:::45:head 2024-09-15T02:48:01.671 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:2b782207:::46:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:7b187fca:::47:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:90ecdf6f:::48:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a5ed95fe:::49:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:ea0eaa55:::50:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f33ef17b:::51:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a0d1b2f6:::52:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:60c5229e:::53:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:edcbc575:::54:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:102cf253:::55:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:efb7fb0b:::56:head 2024-09-15T02:48:01.672 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:50d0a326:::57:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d4dc5daf:::58:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:3a130462:::59:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:ec87ed71:::60:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d5bc9454:::61:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:3ddfe313:::62:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:7c2816b9:::63:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:47e00e4d:::64:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c6410c18:::65:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b48ed237:::66:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:cd63ad31:::67:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b179e92b:::68:head 2024-09-15T02:48:01.673 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:0d9f741a:::69:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6d3352ae:::70:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c6d5c19e:::71:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:bc4729c3:::72:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:77e930b9:::73:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:0abeecfd:::74:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b7c37e15:::75:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b6378398:::76:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:02bd68de:::77:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:cc795d2d:::78:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:630d4fea:::79:head 2024-09-15T02:48:01.674 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:e0d29ef5:::80:head 2024-09-15T02:48:01.675 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:fd6f13d2:::81:head 2024-09-15T02:48:01.675 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:606461d5:::82:head 2024-09-15T02:48:01.675 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:eadbdc43:::83:head 2024-09-15T02:48:01.675 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:8761d0bb:::84:head 2024-09-15T02:48:01.675 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:9ef0186f:::85:head 2024-09-15T02:48:01.675 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:e0d41294:::86:head 2024-09-15T02:48:01.675 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:961de695:::87:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:1423148f:::88:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:633a8fa2:::89:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a8653809:::90:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:3dac8b33:::91:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:35aad435:::92:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f6dcc343:::93:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:dbbdad87:::94:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:1cb48ce0:::95:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:03cd461c:::96:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:17a4ea99:::97:head 2024-09-15T02:48:01.676 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:9993c9a7:::98:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6394211c:::99:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:94c7ae57:::100:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6fdee5bb:::101:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:9a477fd1:::102:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:eb850916:::103:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:affc56b9:::104:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b42dc814:::105:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f319f8f0:::106:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:9a40b9de:::107:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:8b524f28:::108:head 2024-09-15T02:48:01.677 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:e3de589f:::109:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:90f90a5b:::110:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a7b4f1d7:::111:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:af51766e:::112:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b6f90bd1:::113:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:e0261208:::114:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c9569ef7:::115:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:61bebe50:::116:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:fe93412b:::117:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d3d38bee:::118:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:3100ba0c:::119:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d0560ada:::120:head 2024-09-15T02:48:01.678 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f0ea8b35:::121:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:766f231a:::122:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a07a2582:::123:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:bd7c6b3a:::124:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:fb2ddaff:::125:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:4408e1fe:::126:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:ee1df7a7:::127:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c3002909:::128:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:4f48ffa9:::129:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:edf38733:::130:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c08425c0:::131:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:5f902d98:::132:head 2024-09-15T02:48:01.679 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:41ea2c93:::133:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:813cee13:::134:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:0131818d:::135:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:26ba5a85:::136:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:381b8a5a:::137:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:28797e47:::138:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:bfca7f22:::139:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:36807075:::140:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:80b03975:::141:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:5c15709b:::142:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f39ea15e:::143:head 2024-09-15T02:48:01.680 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:ea992956:::144:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:48887b1c:::145:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:9f24a9dd:::146:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:987f100b:::147:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d2dd3581:::148:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:7fed1808:::149:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c80b70e9:::150:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:85ed90f9:::151:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:36428b24:::152:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d044c34a:::153:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:7c18bf58:::154:head 2024-09-15T02:48:01.681 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d1c21232:::155:head 2024-09-15T02:48:01.682 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a7a3c575:::156:head 2024-09-15T02:48:01.682 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:87da0633:::157:head 2024-09-15T02:48:01.682 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d5ac3822:::158:head 2024-09-15T02:48:01.682 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:3f20522d:::159:head 2024-09-15T02:48:01.682 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6ca26563:::160:head 2024-09-15T02:48:01.682 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:532ce135:::161:head 2024-09-15T02:48:01.682 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:c78863e6:::162:head 2024-09-15T02:48:22.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:7462ddf6:::.RoundTripAppendPP (2943 ms) 2024-09-15T02:48:22.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-15T02:48:22.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3010 ms) 2024-09-15T02:48:22.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-15T02:48:22.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3012 ms) 2024-09-15T02:48:22.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-15T02:48:22.392 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3008 ms) 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: setting pool EIO 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: max_success 1770, min_failed 1771 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3978 ms) 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3011 ms) 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (116302 ms total) 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3025 ms) 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-15T02:48:22.393 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8981 ms) 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15002 ms) 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3042 ms) 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30051 ms total) 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2982 ms) 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2982 ms total) 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: 2024-09-15T02:48:22.394 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13978 ms) 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6995 ms) 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7025 ms) 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6988 ms) 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3021 ms) 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-15T02:48:22.395 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6980 ms) 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6996 ms) 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7022 ms) 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6988 ms) 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6999 ms) 2024-09-15T02:48:22.396 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-15T02:48:22.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7037 ms) 2024-09-15T02:48:22.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-15T02:48:22.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6987 ms) 2024-09-15T02:48:22.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-15T02:48:22.397 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6981 ms) 2024-09-15T02:48:56.427 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullP163:head 2024-09-15T02:48:56.428 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:5d165639:::164:head 2024-09-15T02:48:56.428 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f43765fc:::165:head 2024-09-15T02:48:56.428 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b4c720e9:::166:head 2024-09-15T02:48:56.428 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:e694b040:::167:head 2024-09-15T02:48:56.428 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:afa38db2:::168:head 2024-09-15T02:48:56.428 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:77ba9f53:::169:head 2024-09-15T02:48:56.428 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:87495034:::170:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:7c96bf0e:::171:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:dbe346cc:::172:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:e943ec24:::173:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f97a9c0c:::174:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:6f26e74d:::175:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:4f95e106:::176:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:0e6f2f8f:::177:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:05db05f1:::178:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:38a78d66:::179:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:d095610b:::180:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:a1a9d709:::181:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:1e5d39db:::182:head 2024-09-15T02:48:56.429 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:f7df4fb9:::183:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:03a7f161:::184:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:ba70721e:::185:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:28e5662d:::186:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:973d52de:::187:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:4303eb1c:::188:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:b990b48e:::189:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:29b8165b:::190:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:3547f197:::191:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:7e260936:::192:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:1abec7b1:::193:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:10fdda93:::194:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:15817eea:::195:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:770bab57:::196:head 2024-09-15T02:48:56.430 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:ed9e13e7:::197:head 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:71471a8f:::198:head 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: checking for 190:10fb1d02:::199:head 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8032 ms) 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,0 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: first is 1726368491 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,0 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,0 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,0 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,0 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,0 2024-09-15T02:48:56.431 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,1726368496,1726368497,0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,1726368496,1726368497,0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,1726368496,1726368497,0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,1726368496,1726368497,1726368499,1726368500,0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,1726368496,1726368497,1726368499,1726368500,0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368491,1726368493,1726368494,1726368496,1726368497,1726368499,1726368500,0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726368494,1726368496,1726368497,1726368499,1726368500,1726368502,1726368503,0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: first now 1726368494, trimmed 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20966 ms) 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: foo0 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-15T02:48:56.432 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14006 ms) 2024-09-15T02:48:56.433 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-15T02:49:07.419 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ P2 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3027 ms) 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6971 ms) 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6998 ms) 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7028 ms) 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6990 ms) 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-15T02:49:07.420 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6988 ms) 2024-09-15T02:49:07.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-15T02:49:07.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7022 ms) 2024-09-15T02:49:07.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139022 ms total) 2024-09-15T02:49:07.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: 2024-09-15T02:49:07.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-15T02:49:07.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (288357 ms total) 2024-09-15T02:49:07.421 INFO:tasks.workunit.client.0.smithi053.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-15T02:51:19.565 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-09-15T02:51:19.564+0000 7f00a8263640 -1 snap_mapper.add_oid found existing snaps mapped on 58:09612741:test-rados-api-smithi053-38165-76::foo:28, removing 2024-09-15T02:51:19.566 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:51:19.562+0000 7f2264fcc640 -1 snap_mapper.add_oid found existing snaps mapped on 58:09612741:test-rados-api-smithi053-38165-76::foo:28, removing 2024-09-15T02:51:23.588 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:51:23.587+0000 7f9a037cc640 -1 snap_mapper.add_oid found existing snaps mapped on 58:f09808c9:test-rados-api-smithi053-38165-78::foo:29, removing 2024-09-15T02:51:23.589 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:51:23.586+0000 7f225afb8640 -1 snap_mapper.add_oid found existing snaps mapped on 58:f09808c9:test-rados-api-smithi053-38165-78::foo:29, removing 2024-09-15T02:51:48.470 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-09-15T02:51:48.469+0000 7f9a037cc640 -1 snap_mapper.add_oid found existing snaps mapped on 219:b9b9a174:test-rados-api-smithi053-38165-80::foo:2, removing 2024-09-15T02:51:48.471 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:51:48.469+0000 7f225bfba640 -1 snap_mapper.add_oid found existing snaps mapped on 219:b9b9a174:test-rados-api-smithi053-38165-80::foo:2, removing 2024-09-15T02:52:16.469 INFO:tasks.workunit.client.0.smithi053.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18021 ms) 2024-09-15T02:52:16.469 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22021 ms) 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (2949 ms) 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3022 ms) 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3018 ms) 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.470 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (2969 ms) 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4019 ms) 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38060 ms) 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16011 ms) 2024-09-15T02:52:16.471 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4028 ms) 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (2888 ms) 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25096 ms) 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14998 ms) 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-15T02:52:16.472 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5028 ms) 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3889 ms) 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: waiting for scrubs... 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: done waiting 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24102 ms) 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9949 ms) 2024-09-15T02:52:16.473 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-15T02:52:16.474 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.474 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8996 ms) 2024-09-15T02:52:16.474 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-15T02:52:16.474 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:52:16.474 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8999 ms) 2024-09-15T02:53:14.599 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-09-15T02:53:14.598+0000 7f2264fcc640 -1 snap_mapper.add_oid found existing snaps mapped on 58:263f8284:test-rados-api-smithi053-38165-97::foo:31, removing 2024-09-15T02:53:14.600 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-09-15T02:53:14.598+0000 7f00a625f640 -1 snap_mapper.add_oid found existing snaps mapped on 58:263f8284:test-rados-api-smithi053-38165-97::foo:31, removing 2024-09-15T02:55:48.564 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-15T02:55:48.564 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:55:48.564 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8991 ms) 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6028 ms) 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5026 ms) 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25039 ms) 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:55:48.565 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14043 ms) 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11864 ms) 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: 00000012 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12010 ms) 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6034 ms) 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (551176 ms total) 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: 2024-09-15T02:55:48.566 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1285 ms) 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10711 ms) 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17980 ms) 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (29979 ms total) 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-15T02:55:48.567 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7039 ms) 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7991 ms) 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: waiting for scrub... 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: done waiting 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (23976 ms) 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10004 ms) 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-15T02:55:48.568 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8033 ms) 2024-09-15T02:55:48.569 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-15T02:55:48.569 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8006 ms) 2024-09-15T02:55:48.569 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-15T02:55:48.569 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9998 ms) 2024-09-15T02:55:48.569 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-15T02:58:18.550 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8038 ms) 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13057 ms) 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7914 ms) 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13016 ms) 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7983 ms) 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-15T02:58:18.551 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8024 ms) 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8028 ms) 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,0 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: first is 1726369014 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,0 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,0 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,0 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,0 2024-09-15T02:58:18.552 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,0 2024-09-15T02:58:18.553 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,1726369019,1726369020,0 2024-09-15T02:58:18.553 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,1726369019,1726369020,0 2024-09-15T02:58:18.553 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,1726369019,1726369020,0 2024-09-15T02:58:18.553 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,1726369019,1726369020,1726369022,1726369023,0 2024-09-15T02:58:18.553 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,1726369019,1726369020,1726369022,1726369023,0 2024-09-15T02:58:18.553 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369014,1726369016,1726369017,1726369019,1726369020,1726369022,1726369023,0 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: got ls 1726369017,1726369019,1726369020,1726369022,1726369023,1726369025,1726369026,0 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: first now 1726369017, trimmed 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21041 ms) 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: foo0 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13926 ms) 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18048 ms) 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22035 ms) 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-15T02:58:18.554 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4903 ms) 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3034 ms) 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3010 ms) 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (2958 ms) 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (230064 ms total) 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (839362 ms total) 2024-09-15T02:58:18.555 INFO:tasks.workunit.client.0.smithi053.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-15T02:58:18.562 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.562 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37858 2024-09-15T02:58:18.562 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37858 2024-09-15T02:58:18.562 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.562 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38090 2024-09-15T02:58:18.562 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38090 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38646 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38646 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38338 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38338 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.563 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38162 2024-09-15T02:58:18.564 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38162 2024-09-15T02:58:18.564 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.564 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37939 2024-09-15T02:58:18.564 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37939 2024-09-15T02:58:18.564 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.564 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38251 2024-09-15T02:58:18.564 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38251 2024-09-15T02:58:18.565 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.565 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37957 2024-09-15T02:58:18.565 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37957 2024-09-15T02:58:18.565 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.565 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38531 2024-09-15T02:58:18.565 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38531 2024-09-15T02:58:18.566 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.566 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37832 2024-09-15T02:58:18.566 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37832 2024-09-15T02:58:18.566 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.566 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37895 2024-09-15T02:58:18.566 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37895 2024-09-15T02:58:18.567 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.567 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38559 2024-09-15T02:58:18.567 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38559 2024-09-15T02:58:18.567 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.567 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37823 2024-09-15T02:58:18.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37823 2024-09-15T02:58:18.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37843 2024-09-15T02:58:18.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37843 2024-09-15T02:58:18.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38125 2024-09-15T02:58:18.568 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38125 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38223 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38223 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38146 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38146 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38594 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38594 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38504 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38504 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.569 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38026 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38026 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38358 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38358 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37816 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37816 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37875 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37875 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38299 2024-09-15T02:58:18.570 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38299 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=37916 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 37916 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38401 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38401 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ for t in "${!pids[@]}" 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ pid=38456 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ wait 38456 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ exit 0 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ cleanup 2024-09-15T02:58:18.571 INFO:tasks.workunit.client.0.smithi053.stderr:+ pkill -P 37811 2024-09-15T02:58:18.574 INFO:tasks.workunit.client.0.smithi053.stderr:+ true 2024-09-15T02:58:18.574 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:58:18.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:58:18.646 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-09-15T02:58:18.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T02:58:18.974 DEBUG:teuthology.parallel:result is None 2024-09-15T02:58:18.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T02:58:19.001 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T02:58:19.001 DEBUG:teuthology.orchestra.run.smithi053:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:58:19.066 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T02:58:19.066 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T02:58:19.077 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T02:58:19.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:58:19.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.204+0000 7fe399b9d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:19.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.204+0000 7fe399b9d640 10 monclient: build_initial_monmap 2024-09-15T02:58:19.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.204+0000 7fe399b9d640 10 monclient: monmap: 2024-09-15T02:58:19.205 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:19.205 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:19.205201+0000 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:19.205201+0000 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:19.206 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 1 Processor -- start 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 1 -- start start 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient: init 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3940b0aa0 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413b770 0x7fe39413bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe39413b770 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe39413c0e0 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39406d9d0 con 0x7fe39413c0e0 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940b0e70 con 0x7fe3940b0aa0 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3940aca00 con 0x7fe39413b770 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.205+0000 7fe399b9d640 10 monclient(hunting): authenticate will time out at 1635.775944s 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413b770 0x7fe39413bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe393fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe39413c0e0 0x7fe3941384b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59174/0 (socket says 172.21.15.53:59174) 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413b770 0x7fe39413bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56126/0 (socket says 172.21.15.53:56126) 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe393fff640 1 -- 172.21.15.53:0/235364014 learned_addr learned my addr 172.21.15.53:0/235364014 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): get_auth_request con 0x7fe39413b770 auth_method 0 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe3927fc640 1 -- 172.21.15.53:0/235364014 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954146984 0 0) 0x7fe39406d9d0 con 0x7fe39413c0e0 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe3927fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe3927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe3927fc640 10 monclient(hunting): my global_id is 26208 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe3927fc640 1 -- 172.21.15.53:0/235364014 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37c0034e0 con 0x7fe39413c0e0 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient(hunting): handle_auth_done global_id 20306 payload 274 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 1 -- 172.21.15.53:0/235364014 >> v1:172.21.15.104:6789/0 conn(0x7fe3940b0aa0 legacy=0x7fe39406cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 1 -- 172.21.15.53:0/235364014 >> v1:172.21.15.53:6789/0 conn(0x7fe39413c0e0 legacy=0x7fe3941384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 1 monclient: found mon.noname-b 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:19.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.206+0000 7fe392ffd640 1 -- 172.21.15.53:0/235364014 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39406d620 con 0x7fe39413b770 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe392ffd640 1 --2- 172.21.15.53:0/235364014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413b770 0x7fe39413bba0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7fe388009880 tx=0x7fe3880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d38a49cce9e48c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 1 -- 172.21.15.53:0/235364014 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38802ad70 con 0x7fe39413b770 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 10 monclient: dump: 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.219 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.220 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.220 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 10 monclient: _finish_auth 0 2024-09-15T02:58:19.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:19.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe399b9d640 5 monclient: authenticate success, global_id 20306 2024-09-15T02:58:19.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe399b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:19.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 1 -- 172.21.15.53:0/235364014 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe388032440 con 0x7fe39413b770 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 1 -- 172.21.15.53:0/235364014 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3880327c0 con 0x7fe39413b770 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe399b9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe399b9d640 4 set_mon_vals no callback set 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.207+0000 7fe3927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe3927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe3927fc640 10 monclient: dump: 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.221 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 10 monclient: shutdown 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 1 -- 172.21.15.53:0/235364014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413b770 msgr2=0x7fe39413bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 1 --2- 172.21.15.53:0/235364014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413b770 0x7fe39413bba0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7fe388009880 tx=0x7fe3880090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 1 -- 172.21.15.53:0/235364014 shutdown_connections 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 1 --2- 172.21.15.53:0/235364014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413b770 0x7fe39413bba0 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 1 -- 172.21.15.53:0/235364014 >> 172.21.15.53:0/235364014 conn(0x7fe394067b00 msgr2=0x7fe3940b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 1 -- 172.21.15.53:0/235364014 shutdown_connections 2024-09-15T02:58:19.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.208+0000 7fe399b9d640 1 -- 172.21.15.53:0/235364014 wait complete. 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 10 monclient: build_initial_monmap 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 10 monclient: monmap: 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 1 librados: starting msgr at 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 1 librados: starting objecter 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 1 Processor -- start 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 1 -- start start 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 1 librados: setting wanted keys 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 1 librados: calling monclient init 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.209+0000 7fe399b9d640 10 monclient: init 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3940b0aa0 0x7fe39406c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3940b0aa0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39413b770 0x7fe3940a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe39413b770 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:19.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413c0e0 0x7fe39406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe39413c0e0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394119730 con 0x7fe39413b770 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394118270 con 0x7fe3940b0aa0 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe394118790 con 0x7fe39413c0e0 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe399b9d640 10 monclient(hunting): authenticate will time out at 1635.780447s 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39413b770 0x7fe3940a9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413c0e0 0x7fe39406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39413b770 0x7fe3940a9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52300/0 (socket says 172.21.15.53:52300) 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 1 -- 172.21.15.53:0/614028698 learned_addr learned my addr 172.21.15.53:0/614028698 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:19.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): get_auth_request con 0x7fe39413b770 auth_method 0 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 10 monclient(hunting): get_auth_request con 0x7fe39413c0e0 auth_method 0 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3940b0aa0 0x7fe39406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 10 monclient(hunting): get_auth_request con 0x7fe3940b0aa0 auth_method 0 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 10 monclient(hunting): handle_auth_done global_id 26211 payload 824 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 1 -- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413c0e0 msgr2=0x7fe39406a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413c0e0 0x7fe39406a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 1 -- 172.21.15.53:0/614028698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3940b0aa0 msgr2=0x7fe39406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.210+0000 7fe392ffd640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3940b0aa0 0x7fe39406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe392ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:58:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe392ffd640 1 monclient: found mon.noname-a 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe392ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe392ffd640 1 -- 172.21.15.53:0/614028698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388009390 con 0x7fe39413b770 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe393fff640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413c0e0 0x7fe39406a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe392ffd640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39413b770 0x7fe3940a9b00 secure :-1 s=READY pgs=5471 cs=0 l=1 rev1=1 crypto rx=0x7fe388009130 tx=0x7fe388032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 client.?.objecter ms_handle_connect 0x7fe39413b770 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 1 -- 172.21.15.53:0/614028698 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388031730 con 0x7fe39413b770 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: dump: 2024-09-15T02:58:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 5 monclient: authenticate success, global_id 26211 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 1 -- 172.21.15.53:0/614028698 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3880318b0 con 0x7fe39413b770 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 1 -- 172.21.15.53:0/614028698 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388031c70 con 0x7fe39413b770 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe37b7fe640 10 monclient: dump: 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 10 monclient: _renew_subs 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe391ffb640 4 set_mon_vals no callback set 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe394117d50 con 0x7fe39413b770 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 10 client.26211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 10 monclient: _renew_subs 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39406ae00 con 0x7fe39413b770 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.211+0000 7fe399b9d640 1 librados: init done 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.212+0000 7fe399b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.212+0000 7fe399b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.212+0000 7fe399b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.212+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3941184b0 con 0x7fe39413b770 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.213+0000 7fe37b7fe640 1 -- 172.21.15.53:0/614028698 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fe38803c610 con 0x7fe39413b770 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe37b7fe640 1 --2- 172.21.15.53:0/614028698 >> v2:172.21.15.104:6816/32804 conn(0x7fe38006e800 0x7fe380070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe37b7fe640 1 -- 172.21.15.53:0/614028698 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 259..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fe3880ac650 con 0x7fe39413b770 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe37b7fe640 10 client.26211.objecter ms_dispatch 0x7fe394000b90 osd_map(702..702 src has 259..702) v4 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe37b7fe640 3 client.26211.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe37b7fe640 3 client.26211.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe37b7fe640 20 client.26211.objecter dump_active .. 0 homeless 2024-09-15T02:58:19.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe3937fe640 1 --2- 172.21.15.53:0/614028698 >> v2:172.21.15.104:6816/32804 conn(0x7fe38006e800 0x7fe380070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.214+0000 7fe3937fe640 10 monclient: get_auth_request con 0x7fe38006e800 auth_method 0 2024-09-15T02:58:19.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.215+0000 7fe37b7fe640 10 client.26211.objecter ms_handle_connect 0x7fe38006e800 2024-09-15T02:58:19.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.215+0000 7fe3937fe640 1 --2- 172.21.15.53:0/614028698 >> v2:172.21.15.104:6816/32804 conn(0x7fe38006e800 0x7fe380070cb0 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7fe384007fd0 tx=0x7fe384007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.218+0000 7fe37b7fe640 1 -- 172.21.15.53:0/614028698 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe3880769e0 con 0x7fe39413b770 2024-09-15T02:58:19.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.218+0000 7fe37b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.218+0000 7fe37b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:19.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.383+0000 7fe399b9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:58:19.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.383+0000 7fe399b9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:58:19.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.383+0000 7fe399b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:19.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.383+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3941184b0 con 0x7fe39413b770 2024-09-15T02:58:19.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.384+0000 7fe37b7fe640 1 -- 172.21.15.53:0/614028698 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v702) v1 ==== 74+0+14637 (secure 0 0 0) 0x7fe3941184b0 con 0x7fe39413b770 2024-09-15T02:58:19.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.384+0000 7fe37b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:58:19.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.384+0000 7fe37b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:58:19.386 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:58:19.386 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":702,"fsid":"7e7cb607-2b58-4a36-a082-99425792b70f","created":"2024-09-15T02:43:09.455959+0000","modified":"2024-09-15T02:58:18.537916+0000","last_up_change":"2024-09-15T02:43:18.346896+0000","last_in_change":"2024-09-15T02:43:11.511047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":97,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:43:19.373532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:43:20.126006+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-15T02:44:19.495432+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"ab035ba6-bb97-46de-9069-4082e1295127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":33156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":33156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":33156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":33156}]},"public_addr":"172.21.15.53:6801/33156","cluster_addr":"172.21.15.53:6804/33156","heartbeat_back_addr":"172.21.15.53:6809/33156","heartbeat_front_addr":"172.21.15.53:6806/33156","state":["exists","up"]},{"osd":1,"uuid":"eb614e79-5e04-45a4-9962-84c7619ddb87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":33157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":33157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":33157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":33157}]},"public_addr":"172.21.15.53:6800/33157","cluster_addr":"172.21.15.53:6803/33157","heartbeat_back_addr":"172.21.15.53:6811/33157","heartbeat_front_addr":"172.21.15.53:6808/33157","state":["exists","up"]},{"osd":2,"uuid":"08ea49a0-28e1-4740-a4b4-d722560b15c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":33158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":33158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":33158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":33158}]},"public_addr":"172.21.15.53:6802/33158","cluster_addr":"172.21.15.53:6805/33158","heartbeat_back_addr":"172.21.15.53:6810/33158","heartbeat_front_addr":"172.21.15.53:6807/33158","state":["exists","up"]},{"osd":3,"uuid":"1a3a89c3-b449-46ba-bc65-3f53371d3af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":33159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":33159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6815","nonce":33159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":33159}]},"public_addr":"172.21.15.53:6812/33159","cluster_addr":"172.21.15.53:6813/33159","heartbeat_back_addr":"172.21.15.53:6815/33159","heartbeat_front_addr":"172.21.15.53:6814/33159","state":["exists","up"]},{"osd":4,"uuid":"d969fe78-c8c8-4c03-a534-2173406b0ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":33348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":33348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6815","nonce":33348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":33348}]},"public_addr":"172.21.15.104:6812/33348","cluster_addr":"172.21.15.104:6813/33348","heartbeat_back_addr":"172.21.15.104:6815/33348","heartbeat_front_addr":"172.21.15.104:6814/33348","state":["exists","up"]},{"osd":5,"uuid":"51e17a2f-42d3-49a6-b71e-dc7e13877c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":33346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":33346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":33346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":33346}]},"public_addr":"172.21.15.104:6800/33346","cluster_addr":"172.21.15.104:6801/33346","heartbeat_back_addr":"172.21.15.104:6803/33346","heartbeat_front_addr":"172.21.15.104:6802/33346","state":["exists","up"]},{"osd":6,"uuid":"c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":33357}]},"public_addr":"172.21.15.104:6808/33357","cluster_addr":"172.21.15.104:6809/33357","heartbeat_back_addr":"172.21.15.104:6811/33357","heartbeat_front_addr":"172.21.15.104:6810/33357","state":["exists","up"]},{"osd":7,"uuid":"7029aab4-31cc-4219-b7a4-a185564953b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":33347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":33347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":33347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":33347}]},"public_addr":"172.21.15.104:6804/33347","cluster_addr":"172.21.15.104:6805/33347","heartbeat_back_addr":"172.21.15.104:6807/33347","heartbeat_front_addr":"172.21.15.104:6806/33347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.615102+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.701424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.692098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.724924+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.688475+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.598581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.722216+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.661189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":10,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":13,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":54,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":74,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":81,"snaps":[{"begin":2,"length":1}]},{"pool":90,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":91,"snaps":[{"begin":4,"length":4}]},{"pool":102,"snaps":[{"begin":2,"length":1}]},{"pool":108,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":121,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":128,"snaps":[{"begin":12,"length":1}]},{"pool":150,"snaps":[{"begin":14,"length":1}]},{"pool":162,"snaps":[{"begin":17,"length":1}]},{"pool":166,"snaps":[{"begin":19,"length":2}]},{"pool":179,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:58:19.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.386+0000 7fe399b9d640 10 librados: watch_flush enter 2024-09-15T02:58:19.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.386+0000 7fe399b9d640 10 librados: watch_flush exit 2024-09-15T02:58:19.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.386+0000 7fe399b9d640 20 client.26211.objecter shutdown clearing up homeless session... 2024-09-15T02:58:19.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.386+0000 7fe399b9d640 10 client.26211.objecter successfully canceled tick 2024-09-15T02:58:19.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 >> v2:172.21.15.104:6816/32804 conn(0x7fe38006e800 msgr2=0x7fe380070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 --2- 172.21.15.53:0/614028698 >> v2:172.21.15.104:6816/32804 conn(0x7fe38006e800 0x7fe380070cb0 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7fe384007fd0 tx=0x7fe384007e30 comp rx=0 tx=0).stop 2024-09-15T02:58:19.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 10 monclient: shutdown 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39413b770 msgr2=0x7fe3940a9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39413b770 0x7fe3940a9b00 secure :-1 s=READY pgs=5471 cs=0 l=1 rev1=1 crypto rx=0x7fe388009130 tx=0x7fe388032000 comp rx=0 tx=0).stop 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 shutdown_connections 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe39413c0e0 0x7fe39406a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39413b770 0x7fe3940a9b00 unknown :-1 s=CLOSED pgs=5471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 --2- 172.21.15.53:0/614028698 >> v2:172.21.15.104:6816/32804 conn(0x7fe38006e800 0x7fe380070cb0 unknown :-1 s=CLOSED pgs=3685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 --2- 172.21.15.53:0/614028698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3940b0aa0 0x7fe39406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 >> 172.21.15.53:0/614028698 conn(0x7fe394067b00 msgr2=0x7fe39406cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 shutdown_connections 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 -- 172.21.15.53:0/614028698 wait complete. 2024-09-15T02:58:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.387+0000 7fe399b9d640 1 librados: shutdown 2024-09-15T02:58:19.399 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T02:58:19.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:58:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.496+0000 7f11c335d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.496+0000 7f11c335d640 10 monclient: build_initial_monmap 2024-09-15T02:58:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.496+0000 7f11c335d640 10 monclient: monmap: 2024-09-15T02:58:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:19.497481+0000 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:19.497481+0000 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:19.498 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.497+0000 7f11c335d640 1 Processor -- start 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.497+0000 7f11c335d640 1 -- start start 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.497+0000 7f11c335d640 10 monclient: init 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f11bc0b0aa0 0x7f11bc13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient(hunting): picked mon.noname-a con 0x7f11bc0b0aa0 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06e2f0 0x7f11bc06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient(hunting): picked mon.noname-b con 0x7f11bc06e2f0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11bc06ec60 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc13e8a0 con 0x7f11bc06ec60 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc0b0e70 con 0x7f11bc0b0aa0 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc06d660 con 0x7f11bc06e2f0 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c335d640 10 monclient(hunting): authenticate will time out at 1636.068161s 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f11bc0b0aa0 0x7f11bc13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06e2f0 0x7f11bc06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06e2f0 0x7f11bc06e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56144/0 (socket says 172.21.15.53:56144) 2024-09-15T02:58:19.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f11bc0b0aa0 0x7f11bc13dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52302/0 (socket says 172.21.15.53:52302) 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 -- 172.21.15.53:0/2125618842 learned_addr learned my addr 172.21.15.53:0/2125618842 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): get_auth_request con 0x7f11bc06e2f0 auth_method 0 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 10 monclient(hunting): get_auth_request con 0x7f11bc0b0aa0 auth_method 0 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient(hunting): handle_auth_done global_id 20312 payload 274 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 -- 172.21.15.53:0/2125618842 >> v2:172.21.15.53:3300/0 conn(0x7f11bc0b0aa0 msgr2=0x7f11bc13dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 --2- 172.21.15.53:0/2125618842 >> v2:172.21.15.53:3300/0 conn(0x7f11bc0b0aa0 0x7f11bc13dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 -- 172.21.15.53:0/2125618842 >> v1:172.21.15.104:6789/0 conn(0x7f11bc06ec60 legacy=0x7f11bc13a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient: _finish_hunting 0 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 monclient: found mon.noname-b 2024-09-15T02:58:19.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c08f8640 1 -- 172.21.15.53:0/2125618842 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc06d940 con 0x7f11bc06e2f0 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.498+0000 7f11c10f9640 1 --2- 172.21.15.53:0/2125618842 >> v2:172.21.15.53:3300/0 conn(0x7f11bc0b0aa0 0x7f11bc13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11c08f8640 1 --2- 172.21.15.53:0/2125618842 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06e2f0 0x7f11bc06e720 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009910 tx=0x7f11ac0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74661e3a66fe6c0d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 1 -- 172.21.15.53:0/2125618842 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac02ae40 con 0x7f11bc06e2f0 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: dump: 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.506 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: _finish_auth 0 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 1 -- 172.21.15.53:0/2125618842 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac032440 con 0x7f11bc06e2f0 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11c335d640 5 monclient: authenticate success, global_id 20312 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11c335d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 1 -- 172.21.15.53:0/2125618842 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac032820 con 0x7f11bc06e2f0 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11b3fff640 10 monclient: dump: 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11c335d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.499+0000 7f11c335d640 4 set_mon_vals no callback set 2024-09-15T02:58:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 10 monclient: shutdown 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 -- 172.21.15.53:0/2125618842 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06e2f0 msgr2=0x7f11bc06e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 --2- 172.21.15.53:0/2125618842 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06e2f0 0x7f11bc06e720 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009910 tx=0x7f11ac0091d0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 -- 172.21.15.53:0/2125618842 shutdown_connections 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 --2- 172.21.15.53:0/2125618842 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06e2f0 0x7f11bc06e720 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 --2- 172.21.15.53:0/2125618842 >> v2:172.21.15.53:3300/0 conn(0x7f11bc0b0aa0 0x7f11bc13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 -- 172.21.15.53:0/2125618842 >> 172.21.15.53:0/2125618842 conn(0x7f11bc067b00 msgr2=0x7f11bc0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 -- 172.21.15.53:0/2125618842 shutdown_connections 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 1 -- 172.21.15.53:0/2125618842 wait complete. 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 10 monclient: build_initial_monmap 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.500+0000 7f11c335d640 10 monclient: monmap: 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:19.509 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 librados: starting msgr at 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 librados: starting objecter 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 Processor -- start 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 -- start start 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 librados: setting wanted keys 2024-09-15T02:58:19.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 librados: calling monclient init 2024-09-15T02:58:19.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient: init 2024-09-15T02:58:19.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:19.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:58:19.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc06e2f0 0x7f11bc13c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient(hunting): picked mon.noname-b con 0x7f11bc06e2f0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:19.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06ec60 0x7f11bc14de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient(hunting): picked mon.noname-c con 0x7f11bc06ec60 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:19.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11bc0b0aa0 0x7f11bc138660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient(hunting): picked mon.noname-a con 0x7f11bc0b0aa0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc14e830 con 0x7f11bc0b0aa0 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.501+0000 7f11c335d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc13ebd0 con 0x7f11bc06e2f0 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc13ed90 con 0x7f11bc06ec60 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 10 monclient(hunting): authenticate will time out at 1636.072015s 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11bc0b0aa0 0x7f11bc138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06ec60 0x7f11bc14de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06ec60 0x7f11bc14de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56156/0 (socket says 172.21.15.53:56156) 2024-09-15T02:58:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 -- 172.21.15.53:0/992207855 learned_addr learned my addr 172.21.15.53:0/992207855 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11bc0b0aa0 0x7f11bc138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52308/0 (socket says 172.21.15.53:52308) 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): get_auth_request con 0x7f11bc06ec60 auth_method 0 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 10 monclient(hunting): get_auth_request con 0x7f11bc0b0aa0 auth_method 0 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc06e2f0 0x7f11bc13c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 10 monclient(hunting): get_auth_request con 0x7f11bc06e2f0 auth_method 0 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c10f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient(hunting): handle_auth_done global_id 20315 payload 824 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 -- 172.21.15.53:0/992207855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc06e2f0 msgr2=0x7f11bc13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc06e2f0 0x7f11bc13c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 -- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11bc0b0aa0 msgr2=0x7f11bc138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11bc0b0aa0 0x7f11bc138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient: _finish_hunting 0 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 monclient: found mon.noname-c 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 -- 172.21.15.53:0/992207855 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b800b630 con 0x7f11bc06ec60 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c18fa640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11bc0b0aa0 0x7f11bc138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c08f8640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06ec60 0x7f11bc14de40 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7f11ac035040 tx=0x7f11ac032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 client.?.objecter ms_handle_connect 0x7f11bc06ec60 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 1 -- 172.21.15.53:0/992207855 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac031bb0 con 0x7f11bc06ec60 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: dump: 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: _finish_auth 0 2024-09-15T02:58:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 5 monclient: authenticate success, global_id 20315 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 1 -- 172.21.15.53:0/992207855 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac031d30 con 0x7f11bc06ec60 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 1 -- 172.21.15.53:0/992207855 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac031ea0 con 0x7f11bc06ec60 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b17fa640 10 monclient: dump: 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.521 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11b37fe640 4 set_mon_vals no callback set 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 10 monclient: _renew_subs 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.502+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac009450 con 0x7f11bc06ec60 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.503+0000 7f11c335d640 10 client.20315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.503+0000 7f11c335d640 10 monclient: _renew_subs 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.503+0000 7f11c335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.503+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc14e520 con 0x7f11bc06ec60 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.503+0000 7f11c335d640 1 librados: init done 2024-09-15T02:58:19.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.504+0000 7f11c335d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.504+0000 7f11c335d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.504+0000 7f11c335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.504+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc0646e0 con 0x7f11bc06ec60 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.504+0000 7f11b17fa640 1 -- 172.21.15.53:0/992207855 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f11ac039040 con 0x7f11bc06ec60 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11b17fa640 1 --2- 172.21.15.53:0/992207855 >> v2:172.21.15.104:6816/32804 conn(0x7f11a006e800 0x7f11a0070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11b17fa640 1 -- 172.21.15.53:0/992207855 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 259..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f11ac02ae40 con 0x7f11bc06ec60 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11b17fa640 10 client.20315.objecter ms_dispatch 0x7f11bc000b90 osd_map(702..702 src has 259..702) v4 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11b17fa640 3 client.20315.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11b17fa640 3 client.20315.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11b17fa640 20 client.20315.objecter dump_active .. 0 homeless 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11c10f9640 1 --2- 172.21.15.53:0/992207855 >> v2:172.21.15.104:6816/32804 conn(0x7f11a006e800 0x7f11a0070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.505+0000 7f11c10f9640 10 monclient: get_auth_request con 0x7f11a006e800 auth_method 0 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.506+0000 7f11b17fa640 10 client.20315.objecter ms_handle_connect 0x7f11a006e800 2024-09-15T02:58:19.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.506+0000 7f11c10f9640 1 --2- 172.21.15.53:0/992207855 >> v2:172.21.15.104:6816/32804 conn(0x7f11a006e800 0x7f11a0070cb0 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7f11b800bf00 tx=0x7f11b800b3b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.509+0000 7f11b17fa640 1 -- 172.21.15.53:0/992207855 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f11ac075b40 con 0x7f11bc06ec60 2024-09-15T02:58:19.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.509+0000 7f11b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.509+0000 7f11b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:19.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.685+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11bc1383c0 con 0x7f11a006e800 2024-09-15T02:58:19.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.690+0000 7f11b17fa640 1 -- 172.21.15.53:0/992207855 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+153796 (secure 0 0 0) 0x7f11bc1383c0 con 0x7f11a006e800 2024-09-15T02:58:19.693 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:58:19.694 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:58:19.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 10 librados: watch_flush enter 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 10 librados: watch_flush exit 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 20 client.20315.objecter shutdown clearing up homeless session... 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 10 client.20315.objecter successfully canceled tick 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 >> v2:172.21.15.104:6816/32804 conn(0x7f11a006e800 msgr2=0x7f11a0070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 --2- 172.21.15.53:0/992207855 >> v2:172.21.15.104:6816/32804 conn(0x7f11a006e800 0x7f11a0070cb0 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7f11b800bf00 tx=0x7f11b800b3b0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 10 monclient: shutdown 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06ec60 msgr2=0x7f11bc14de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06ec60 0x7f11bc14de40 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7f11ac035040 tx=0x7f11ac032000 comp rx=0 tx=0).stop 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 shutdown_connections 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11bc0b0aa0 0x7f11bc138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11bc06ec60 0x7f11bc14de40 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 --2- 172.21.15.53:0/992207855 >> v2:172.21.15.104:6816/32804 conn(0x7f11a006e800 0x7f11a0070cb0 unknown :-1 s=CLOSED pgs=3686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 --2- 172.21.15.53:0/992207855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc06e2f0 0x7f11bc13c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 >> 172.21.15.53:0/992207855 conn(0x7f11bc067b00 msgr2=0x7f11bc1379c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:19.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 shutdown_connections 2024-09-15T02:58:19.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 -- 172.21.15.53:0/992207855 wait complete. 2024-09-15T02:58:19.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.693+0000 7f11c335d640 1 librados: shutdown 2024-09-15T02:58:19.705 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":1145,"stamp":"2024-09-15T02:58:19.540071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2471344,"kb_used_data":50876,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752503376,"statfs":{"total":773094113280,"available":770563457024,"internally_reserved":0,"allocated":52097024,"data_stored":25969454,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005021"},"pg_stats":[{"pgid":"12.10","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539924+0000","last_change":"2024-09-15T02:44:21.592595+0000","last_active":"2024-09-15T02:58:16.539924+0000","last_peered":"2024-09-15T02:58:16.539924+0000","last_clean":"2024-09-15T02:58:16.539924+0000","last_became_active":"2024-09-15T02:44:21.589224+0000","last_became_peered":"2024-09-15T02:44:21.589224+0000","last_unstale":"2024-09-15T02:58:16.539924+0000","last_undegraded":"2024-09-15T02:58:16.539924+0000","last_fullsized":"2024-09-15T02:58:16.539924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:30:34.422407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836595+0000","last_change":"2024-09-15T02:44:21.472953+0000","last_active":"2024-09-15T02:58:18.836595+0000","last_peered":"2024-09-15T02:58:18.836595+0000","last_clean":"2024-09-15T02:58:18.836595+0000","last_became_active":"2024-09-15T02:44:21.472608+0000","last_became_peered":"2024-09-15T02:44:21.472608+0000","last_unstale":"2024-09-15T02:58:18.836595+0000","last_undegraded":"2024-09-15T02:58:18.836595+0000","last_fullsized":"2024-09-15T02:58:18.836595+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:48.488509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.593941+0000","last_change":"2024-09-15T02:44:21.605637+0000","last_active":"2024-09-15T02:58:13.593941+0000","last_peered":"2024-09-15T02:58:13.593941+0000","last_clean":"2024-09-15T02:58:13.593941+0000","last_became_active":"2024-09-15T02:44:21.604946+0000","last_became_peered":"2024-09-15T02:44:21.604946+0000","last_unstale":"2024-09-15T02:58:13.593941+0000","last_undegraded":"2024-09-15T02:58:13.593941+0000","last_fullsized":"2024-09-15T02:58:13.593941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:21:30.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.538478+0000","last_change":"2024-09-15T02:44:21.631801+0000","last_active":"2024-09-15T02:58:15.538478+0000","last_peered":"2024-09-15T02:58:15.538478+0000","last_clean":"2024-09-15T02:58:15.538478+0000","last_became_active":"2024-09-15T02:44:21.621686+0000","last_became_peered":"2024-09-15T02:44:21.621686+0000","last_unstale":"2024-09-15T02:58:15.538478+0000","last_undegraded":"2024-09-15T02:58:15.538478+0000","last_fullsized":"2024-09-15T02:58:15.538478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:21:17.555939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836680+0000","last_change":"2024-09-15T02:44:21.559633+0000","last_active":"2024-09-15T02:58:18.836680+0000","last_peered":"2024-09-15T02:58:18.836680+0000","last_clean":"2024-09-15T02:58:18.836680+0000","last_became_active":"2024-09-15T02:44:21.558951+0000","last_became_peered":"2024-09-15T02:44:21.558951+0000","last_unstale":"2024-09-15T02:58:18.836680+0000","last_undegraded":"2024-09-15T02:58:18.836680+0000","last_fullsized":"2024-09-15T02:58:18.836680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:27.075112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236232+0000","last_change":"2024-09-15T02:44:21.593138+0000","last_active":"2024-09-15T02:58:17.236232+0000","last_peered":"2024-09-15T02:58:17.236232+0000","last_clean":"2024-09-15T02:58:17.236232+0000","last_became_active":"2024-09-15T02:44:21.592665+0000","last_became_peered":"2024-09-15T02:44:21.592665+0000","last_unstale":"2024-09-15T02:58:17.236232+0000","last_undegraded":"2024-09-15T02:58:17.236232+0000","last_fullsized":"2024-09-15T02:58:17.236232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:20.534135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235717+0000","last_change":"2024-09-15T02:44:21.537510+0000","last_active":"2024-09-15T02:58:17.235717+0000","last_peered":"2024-09-15T02:58:17.235717+0000","last_clean":"2024-09-15T02:58:17.235717+0000","last_became_active":"2024-09-15T02:44:21.537279+0000","last_became_peered":"2024-09-15T02:44:21.537279+0000","last_unstale":"2024-09-15T02:58:17.235717+0000","last_undegraded":"2024-09-15T02:58:17.235717+0000","last_fullsized":"2024-09-15T02:58:17.235717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:55:56.290343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235427+0000","last_change":"2024-09-15T02:44:21.613091+0000","last_active":"2024-09-15T02:58:17.235427+0000","last_peered":"2024-09-15T02:58:17.235427+0000","last_clean":"2024-09-15T02:58:17.235427+0000","last_became_active":"2024-09-15T02:44:21.612841+0000","last_became_peered":"2024-09-15T02:44:21.612841+0000","last_unstale":"2024-09-15T02:58:17.235427+0000","last_undegraded":"2024-09-15T02:58:17.235427+0000","last_fullsized":"2024-09-15T02:58:17.235427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:28:24.283127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537152+0000","last_change":"2024-09-15T02:44:21.543344+0000","last_active":"2024-09-15T02:58:15.537152+0000","last_peered":"2024-09-15T02:58:15.537152+0000","last_clean":"2024-09-15T02:58:15.537152+0000","last_became_active":"2024-09-15T02:44:21.522860+0000","last_became_peered":"2024-09-15T02:44:21.522860+0000","last_unstale":"2024-09-15T02:58:15.537152+0000","last_undegraded":"2024-09-15T02:58:15.537152+0000","last_fullsized":"2024-09-15T02:58:15.537152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:30.830504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541359+0000","last_change":"2024-09-15T02:44:21.538578+0000","last_active":"2024-09-15T02:58:16.541359+0000","last_peered":"2024-09-15T02:58:16.541359+0000","last_clean":"2024-09-15T02:58:16.541359+0000","last_became_active":"2024-09-15T02:44:21.535334+0000","last_became_peered":"2024-09-15T02:44:21.535334+0000","last_unstale":"2024-09-15T02:58:16.541359+0000","last_undegraded":"2024-09-15T02:58:16.541359+0000","last_fullsized":"2024-09-15T02:58:16.541359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:38:25.943125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542872+0000","last_change":"2024-09-15T02:44:21.614405+0000","last_active":"2024-09-15T02:58:17.542872+0000","last_peered":"2024-09-15T02:58:17.542872+0000","last_clean":"2024-09-15T02:58:17.542872+0000","last_became_active":"2024-09-15T02:44:21.614186+0000","last_became_peered":"2024-09-15T02:44:21.614186+0000","last_unstale":"2024-09-15T02:58:17.542872+0000","last_undegraded":"2024-09-15T02:58:17.542872+0000","last_fullsized":"2024-09-15T02:58:17.542872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:52.698525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542738+0000","last_change":"2024-09-15T02:44:21.527859+0000","last_active":"2024-09-15T02:58:17.542738+0000","last_peered":"2024-09-15T02:58:17.542738+0000","last_clean":"2024-09-15T02:58:17.542738+0000","last_became_active":"2024-09-15T02:44:21.527331+0000","last_became_peered":"2024-09-15T02:44:21.527331+0000","last_unstale":"2024-09-15T02:58:17.542738+0000","last_undegraded":"2024-09-15T02:58:17.542738+0000","last_fullsized":"2024-09-15T02:58:17.542738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:55:15.776872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":653,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235679+0000","last_change":"2024-09-15T02:43:23.626724+0000","last_active":"2024-09-15T02:58:17.235679+0000","last_peered":"2024-09-15T02:58:17.235679+0000","last_clean":"2024-09-15T02:58:17.235679+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:58:17.235679+0000","last_undegraded":"2024-09-15T02:58:17.235679+0000","last_fullsized":"2024-09-15T02:58:17.235679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:48.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540373+0000","last_change":"2024-09-15T02:44:21.539798+0000","last_active":"2024-09-15T02:58:17.540373+0000","last_peered":"2024-09-15T02:58:17.540373+0000","last_clean":"2024-09-15T02:58:17.540373+0000","last_became_active":"2024-09-15T02:44:21.530274+0000","last_became_peered":"2024-09-15T02:44:21.530274+0000","last_unstale":"2024-09-15T02:58:17.540373+0000","last_undegraded":"2024-09-15T02:58:17.540373+0000","last_fullsized":"2024-09-15T02:58:17.540373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:24:08.051420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":655,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537819+0000","last_change":"2024-09-15T02:43:23.426048+0000","last_active":"2024-09-15T02:58:15.537819+0000","last_peered":"2024-09-15T02:58:15.537819+0000","last_clean":"2024-09-15T02:58:15.537819+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:58:15.537819+0000","last_undegraded":"2024-09-15T02:58:15.537819+0000","last_fullsized":"2024-09-15T02:58:15.537819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:19.460873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.594062+0000","last_change":"2024-09-15T02:44:21.579868+0000","last_active":"2024-09-15T02:58:13.594062+0000","last_peered":"2024-09-15T02:58:13.594062+0000","last_clean":"2024-09-15T02:58:13.594062+0000","last_became_active":"2024-09-15T02:44:21.579154+0000","last_became_peered":"2024-09-15T02:44:21.579154+0000","last_unstale":"2024-09-15T02:58:13.594062+0000","last_undegraded":"2024-09-15T02:58:13.594062+0000","last_fullsized":"2024-09-15T02:58:13.594062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:45:09.149484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":650,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914260+0000","last_change":"2024-09-15T02:43:23.420483+0000","last_active":"2024-09-15T02:58:16.914260+0000","last_peered":"2024-09-15T02:58:16.914260+0000","last_clean":"2024-09-15T02:58:16.914260+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:58:16.914260+0000","last_undegraded":"2024-09-15T02:58:16.914260+0000","last_fullsized":"2024-09-15T02:58:16.914260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:03:14.233618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.594285+0000","last_change":"2024-09-15T02:44:21.537181+0000","last_active":"2024-09-15T02:58:13.594285+0000","last_peered":"2024-09-15T02:58:13.594285+0000","last_clean":"2024-09-15T02:58:13.594285+0000","last_became_active":"2024-09-15T02:44:21.533501+0000","last_became_peered":"2024-09-15T02:44:21.533501+0000","last_unstale":"2024-09-15T02:58:13.594285+0000","last_undegraded":"2024-09-15T02:58:13.594285+0000","last_fullsized":"2024-09-15T02:58:13.594285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:42.258617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":655,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537074+0000","last_change":"2024-09-15T02:43:23.426071+0000","last_active":"2024-09-15T02:58:15.537074+0000","last_peered":"2024-09-15T02:58:15.537074+0000","last_clean":"2024-09-15T02:58:15.537074+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:58:15.537074+0000","last_undegraded":"2024-09-15T02:58:15.537074+0000","last_fullsized":"2024-09-15T02:58:15.537074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:00:03.954753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540360+0000","last_change":"2024-09-15T02:44:21.561913+0000","last_active":"2024-09-15T02:58:17.540360+0000","last_peered":"2024-09-15T02:58:17.540360+0000","last_clean":"2024-09-15T02:58:17.540360+0000","last_became_active":"2024-09-15T02:44:21.560425+0000","last_became_peered":"2024-09-15T02:44:21.560425+0000","last_unstale":"2024-09-15T02:58:17.540360+0000","last_undegraded":"2024-09-15T02:58:17.540360+0000","last_fullsized":"2024-09-15T02:58:17.540360+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:40:57.301643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":658,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836590+0000","last_change":"2024-09-15T02:43:23.423983+0000","last_active":"2024-09-15T02:58:18.836590+0000","last_peered":"2024-09-15T02:58:18.836590+0000","last_clean":"2024-09-15T02:58:18.836590+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:58:18.836590+0000","last_undegraded":"2024-09-15T02:58:18.836590+0000","last_fullsized":"2024-09-15T02:58:18.836590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:45.683030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539798+0000","last_change":"2024-09-15T02:44:21.500932+0000","last_active":"2024-09-15T02:58:16.539798+0000","last_peered":"2024-09-15T02:58:16.539798+0000","last_clean":"2024-09-15T02:58:16.539798+0000","last_became_active":"2024-09-15T02:44:21.500646+0000","last_became_peered":"2024-09-15T02:44:21.500646+0000","last_unstale":"2024-09-15T02:58:16.539798+0000","last_undegraded":"2024-09-15T02:58:16.539798+0000","last_fullsized":"2024-09-15T02:58:16.539798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:08.733289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":663,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540358+0000","last_change":"2024-09-15T02:43:23.419938+0000","last_active":"2024-09-15T02:58:17.540358+0000","last_peered":"2024-09-15T02:58:17.540358+0000","last_clean":"2024-09-15T02:58:17.540358+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:58:17.540358+0000","last_undegraded":"2024-09-15T02:58:17.540358+0000","last_fullsized":"2024-09-15T02:58:17.540358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:09:35.753407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235268+0000","last_change":"2024-09-15T02:44:21.504909+0000","last_active":"2024-09-15T02:58:17.235268+0000","last_peered":"2024-09-15T02:58:17.235268+0000","last_clean":"2024-09-15T02:58:17.235268+0000","last_became_active":"2024-09-15T02:44:21.504665+0000","last_became_peered":"2024-09-15T02:44:21.504665+0000","last_unstale":"2024-09-15T02:58:17.235268+0000","last_undegraded":"2024-09-15T02:58:17.235268+0000","last_fullsized":"2024-09-15T02:58:17.235268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:17:58.326032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":649,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913924+0000","last_change":"2024-09-15T02:43:23.420430+0000","last_active":"2024-09-15T02:58:16.913924+0000","last_peered":"2024-09-15T02:58:16.913924+0000","last_clean":"2024-09-15T02:58:16.913924+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:58:16.913924+0000","last_undegraded":"2024-09-15T02:58:16.913924+0000","last_fullsized":"2024-09-15T02:58:16.913924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:05:17.337850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.543353+0000","last_change":"2024-09-15T02:44:21.645181+0000","last_active":"2024-09-15T02:58:16.543353+0000","last_peered":"2024-09-15T02:58:16.543353+0000","last_clean":"2024-09-15T02:58:16.543353+0000","last_became_active":"2024-09-15T02:44:21.644939+0000","last_became_peered":"2024-09-15T02:44:21.644939+0000","last_unstale":"2024-09-15T02:58:16.543353+0000","last_undegraded":"2024-09-15T02:58:16.543353+0000","last_fullsized":"2024-09-15T02:58:16.543353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:22:20.773021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":657,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836538+0000","last_change":"2024-09-15T02:43:23.423477+0000","last_active":"2024-09-15T02:58:18.836538+0000","last_peered":"2024-09-15T02:58:18.836538+0000","last_clean":"2024-09-15T02:58:18.836538+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:58:18.836538+0000","last_undegraded":"2024-09-15T02:58:18.836538+0000","last_fullsized":"2024-09-15T02:58:18.836538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:05:30.605263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":735,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913790+0000","last_change":"2024-09-15T02:43:20.387731+0000","last_active":"2024-09-15T02:58:16.913790+0000","last_peered":"2024-09-15T02:58:16.913790+0000","last_clean":"2024-09-15T02:58:16.913790+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:58:16.913790+0000","last_undegraded":"2024-09-15T02:58:16.913790+0000","last_fullsized":"2024-09-15T02:58:16.913790+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_clean_scrub_stamp":"2024-09-15T02:43:19.374225+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:33:29.816927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":639,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914166+0000","last_change":"2024-09-15T02:44:21.629491+0000","last_active":"2024-09-15T02:58:16.914166+0000","last_peered":"2024-09-15T02:58:16.914166+0000","last_clean":"2024-09-15T02:58:16.914166+0000","last_became_active":"2024-09-15T02:44:21.629239+0000","last_became_peered":"2024-09-15T02:44:21.629239+0000","last_unstale":"2024-09-15T02:58:16.914166+0000","last_undegraded":"2024-09-15T02:58:16.914166+0000","last_fullsized":"2024-09-15T02:58:16.914166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:56.354264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.538060+0000","last_change":"2024-09-15T02:44:21.573593+0000","last_active":"2024-09-15T02:58:15.538060+0000","last_peered":"2024-09-15T02:58:15.538060+0000","last_clean":"2024-09-15T02:58:15.538060+0000","last_became_active":"2024-09-15T02:44:21.571415+0000","last_became_peered":"2024-09-15T02:44:21.571415+0000","last_unstale":"2024-09-15T02:58:15.538060+0000","last_undegraded":"2024-09-15T02:58:15.538060+0000","last_fullsized":"2024-09-15T02:58:15.538060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:12.880270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537934+0000","last_change":"2024-09-15T02:44:21.629604+0000","last_active":"2024-09-15T02:58:15.537934+0000","last_peered":"2024-09-15T02:58:15.537934+0000","last_clean":"2024-09-15T02:58:15.537934+0000","last_became_active":"2024-09-15T02:44:21.628817+0000","last_became_peered":"2024-09-15T02:44:21.628817+0000","last_unstale":"2024-09-15T02:58:15.537934+0000","last_undegraded":"2024-09-15T02:58:15.537934+0000","last_fullsized":"2024-09-15T02:58:15.537934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:52:41.637023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236078+0000","last_change":"2024-09-15T02:44:21.583579+0000","last_active":"2024-09-15T02:58:17.236078+0000","last_peered":"2024-09-15T02:58:17.236078+0000","last_clean":"2024-09-15T02:58:17.236078+0000","last_became_active":"2024-09-15T02:44:21.583340+0000","last_became_peered":"2024-09-15T02:44:21.583340+0000","last_unstale":"2024-09-15T02:58:17.236078+0000","last_undegraded":"2024-09-15T02:58:17.236078+0000","last_fullsized":"2024-09-15T02:58:17.236078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:01:58.119370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235646+0000","last_change":"2024-09-15T02:44:21.616190+0000","last_active":"2024-09-15T02:58:17.235646+0000","last_peered":"2024-09-15T02:58:17.235646+0000","last_clean":"2024-09-15T02:58:17.235646+0000","last_became_active":"2024-09-15T02:44:21.615991+0000","last_became_peered":"2024-09-15T02:44:21.615991+0000","last_unstale":"2024-09-15T02:58:17.235646+0000","last_undegraded":"2024-09-15T02:58:17.235646+0000","last_fullsized":"2024-09-15T02:58:17.235646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:25:27.900125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539926+0000","last_change":"2024-09-15T02:44:21.608529+0000","last_active":"2024-09-15T02:58:16.539926+0000","last_peered":"2024-09-15T02:58:16.539926+0000","last_clean":"2024-09-15T02:58:16.539926+0000","last_became_active":"2024-09-15T02:44:21.588781+0000","last_became_peered":"2024-09-15T02:44:21.588781+0000","last_unstale":"2024-09-15T02:58:16.539926+0000","last_undegraded":"2024-09-15T02:58:16.539926+0000","last_fullsized":"2024-09-15T02:58:16.539926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:49.754802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.614367+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.613899+0000","last_became_peered":"2024-09-15T02:44:21.613899+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:18:47.868539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1","version":"22'7","reported_seq":655,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235332+0000","last_change":"2024-09-15T02:44:21.595390+0000","last_active":"2024-09-15T02:58:17.235332+0000","last_peered":"2024-09-15T02:58:17.235332+0000","last_clean":"2024-09-15T02:58:17.235332+0000","last_became_active":"2024-09-15T02:44:21.594670+0000","last_became_peered":"2024-09-15T02:44:21.594670+0000","last_unstale":"2024-09-15T02:58:17.235332+0000","last_undegraded":"2024-09-15T02:58:17.235332+0000","last_fullsized":"2024-09-15T02:58:17.235332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:52:07.118143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.536146+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.535920+0000","last_became_peered":"2024-09-15T02:44:21.535920+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.095882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235267+0000","last_change":"2024-09-15T02:44:21.578890+0000","last_active":"2024-09-15T02:58:17.235267+0000","last_peered":"2024-09-15T02:58:17.235267+0000","last_clean":"2024-09-15T02:58:17.235267+0000","last_became_active":"2024-09-15T02:44:21.578571+0000","last_became_peered":"2024-09-15T02:44:21.578571+0000","last_unstale":"2024-09-15T02:58:17.235267+0000","last_undegraded":"2024-09-15T02:58:17.235267+0000","last_fullsized":"2024-09-15T02:58:17.235267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:01.395702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542928+0000","last_change":"2024-09-15T02:44:21.556333+0000","last_active":"2024-09-15T02:58:17.542928+0000","last_peered":"2024-09-15T02:58:17.542928+0000","last_clean":"2024-09-15T02:58:17.542928+0000","last_became_active":"2024-09-15T02:44:21.556078+0000","last_became_peered":"2024-09-15T02:44:21.556078+0000","last_unstale":"2024-09-15T02:58:17.542928+0000","last_undegraded":"2024-09-15T02:58:17.542928+0000","last_fullsized":"2024-09-15T02:58:17.542928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:35:40.520618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541145+0000","last_change":"2024-09-15T02:44:21.583194+0000","last_active":"2024-09-15T02:58:16.541145+0000","last_peered":"2024-09-15T02:58:16.541145+0000","last_clean":"2024-09-15T02:58:16.541145+0000","last_became_active":"2024-09-15T02:44:21.582718+0000","last_became_peered":"2024-09-15T02:44:21.582718+0000","last_unstale":"2024-09-15T02:58:16.541145+0000","last_undegraded":"2024-09-15T02:58:16.541145+0000","last_fullsized":"2024-09-15T02:58:16.541145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:48:37.528157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.540867+0000","last_change":"2024-09-15T02:44:21.584252+0000","last_active":"2024-09-15T02:58:16.540867+0000","last_peered":"2024-09-15T02:58:16.540867+0000","last_clean":"2024-09-15T02:58:16.540867+0000","last_became_active":"2024-09-15T02:44:21.584028+0000","last_became_peered":"2024-09-15T02:44:21.584028+0000","last_unstale":"2024-09-15T02:58:16.540867+0000","last_undegraded":"2024-09-15T02:58:16.540867+0000","last_fullsized":"2024-09-15T02:58:16.540867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:14.221941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575031,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312068,"kb_used_data":6388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059772,"statfs":{"total":96636764160,"available":96317206528,"internally_reserved":0,"allocated":6541312,"data_stored":3635868,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.69399999999999995,"5min":0.75,"15min":1.3959999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.28100000000000003},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.80500000000000005},"last":0.76500000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.55300000000000005},"min":{"1min":0.46100000000000002,"5min":0.33600000000000002,"15min":0.23400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.94599999999999995},"last":0.745},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59799999999999998,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.27300000000000002},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":1.4430000000000001},"last":0.67900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60799999999999998,"15min":0.56200000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.27000000000000002},"max":{"1min":0.745,"5min":0.78600000000000003,"15min":0.95899999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.20300000000000001},"max":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":1.478},"last":0.80300000000000005}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57599999999999996,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.28299999999999997},"max":{"1min":0.72699999999999998,"5min":0.83699999999999997,"15min":2.4300000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.57299999999999995},"min":{"1min":0.502,"5min":0.40600000000000003,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.89700000000000002,"15min":1.6120000000000001},"last":0.71599999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45100000000000001,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.24199999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.755},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.28000000000000003},"max":{"1min":0.70599999999999996,"5min":0.83399999999999996,"15min":0.95899999999999996},"last":0.626}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":0.45400000000000001},"min":{"1min":0.35499999999999998,"5min":0.30099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72899999999999998,"15min":0.88900000000000001},"last":0.502},{"interface":"front","average":{"1min":0.44,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":2.2400000000000002},"last":0.57699999999999996}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.77600000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.52500000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.79300000000000004,"5min":0.89700000000000002,"15min":0.92800000000000005},"last":0.55000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312012,"kb_used_data":6340,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059828,"statfs":{"total":96636764160,"available":96317263872,"internally_reserved":0,"allocated":6492160,"data_stored":3106945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.34599999999999997,"15min":0.27000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":5.0250000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.874},"min":{"1min":0.46600000000000003,"5min":0.37,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":0.85699999999999998,"15min":71.516999999999996},"last":0.45400000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.94699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.25900000000000001},"max":{"1min":0.69399999999999995,"5min":0.84199999999999997,"15min":93.381},"last":0.5},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56100000000000005,"15min":0.86399999999999999},"min":{"1min":0.39300000000000002,"5min":0.33500000000000002,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.90900000000000003,"15min":79.905000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.66400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.93899999999999995,"15min":29.305},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.318},"max":{"1min":0.66500000000000004,"5min":0.85499999999999998,"15min":9.8510000000000009},"last":0.54800000000000004}]},{"osd":3,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.69499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.255},"max":{"1min":0.75900000000000001,"5min":0.96599999999999997,"15min":34.877000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.29299999999999998},"max":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":31.657},"last":0.51000000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.51000000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.20300000000000001},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":20.212},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.442,"15min":0.46300000000000002},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.64700000000000002,"5min":0.71599999999999997,"15min":5.6909999999999998},"last":0.35999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.215},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80000000000000004},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.248},"max":{"1min":0.71499999999999997,"5min":0.81799999999999995,"15min":3.012},"last":0.58699999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47099999999999997,"15min":0.59499999999999997},"min":{"1min":0.35099999999999998,"5min":0.32500000000000001,"15min":0.24299999999999999},"max":{"1min":0.68999999999999995,"5min":0.71299999999999997,"15min":28.756},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.48899999999999999,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.32000000000000001,"15min":0.249},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":27.602},"last":0.436}]}]},{"osd":5,"up_from":13,"seq":55834575032,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":6352,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":6504448,"data_stored":3112540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.69899999999999995},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":17.704999999999998},"last":0.435},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.621,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.78100000000000003,"5min":0.872,"15min":17.465},"last":0.54900000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.22900000000000001},"max":{"1min":0.81999999999999995,"5min":0.83999999999999997,"15min":7.1710000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.42999999999999999,"5min":0.33300000000000002,"15min":0.27700000000000002},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":6.7999999999999998},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1000000000000001,"15min":8.2789999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.60299999999999998},"min":{"1min":0.51600000000000001,"5min":0.33800000000000002,"15min":0.29099999999999998},"max":{"1min":0.79300000000000004,"5min":1.133,"15min":3.0059999999999998},"last":0.61299999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.628},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.25700000000000001},"max":{"1min":0.81299999999999994,"5min":0.872,"15min":7.8090000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63600000000000001,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.38200000000000001,"15min":0.28599999999999998},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":8.6829999999999998},"last":0.72399999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46300000000000002,"15min":0.44600000000000001},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.216},"max":{"1min":0.68100000000000005,"5min":0.82099999999999995,"15min":0.98699999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.23000000000000001},"max":{"1min":0.68300000000000005,"5min":0.78000000000000003,"15min":2.0750000000000002},"last":0.63200000000000001}]},{"osd":6,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.22800000000000001},"max":{"1min":0.67000000000000004,"5min":0.81799999999999995,"15min":3.4420000000000002},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.39100000000000001,"5min":0.33300000000000002,"15min":0.25},"max":{"1min":0.68899999999999995,"5min":0.86299999999999999,"15min":3.0510000000000002},"last":0.65200000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.253},"max":{"1min":0.69699999999999995,"5min":0.79400000000000004,"15min":2.9870000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.94399999999999995,"15min":1.337},"last":0.78200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307884,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063956,"statfs":{"total":96636764160,"available":96321490944,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.86299999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.316},"max":{"1min":0.72099999999999997,"5min":0.873,"15min":0.93100000000000005},"last":0.72699999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.69699999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.32800000000000001},"max":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.379,"15min":0.223},"max":{"1min":0.73699999999999999,"5min":0.89400000000000002,"15min":0.94199999999999995},"last":0.754},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.83499999999999996}]},{"osd":3,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.47799999999999998,"5min":0.36699999999999999,"15min":0.26800000000000002},"max":{"1min":0.82499999999999996,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60399999999999998,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.36099999999999999},"max":{"1min":0.752,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.23200000000000001},"max":{"1min":0.67400000000000004,"5min":0.69699999999999995,"15min":0.81599999999999995},"last":0.47999999999999998}]},{"osd":6,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.497},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42099999999999999,"15min":0.44700000000000001},"min":{"1min":0.316,"5min":0.31,"15min":0.251},"max":{"1min":0.505,"5min":0.65700000000000003,"15min":0.69699999999999995},"last":0.59599999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.78600000000000003,"5min":1.48,"15min":1.48},"last":0.439},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.49199999999999999,"15min":0.50600000000000001},"min":{"1min":0.373,"5min":0.31900000000000001,"15min":0.28000000000000003},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.81799999999999995},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575031,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307852,"kb_used_data":6336,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063988,"statfs":{"total":96636764160,"available":96321523712,"internally_reserved":0,"allocated":6488064,"data_stored":3127885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.371,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.52200000000000002,"5min":0.72399999999999998,"15min":29.562999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47899999999999998,"15min":0.44400000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":0.88600000000000001,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":0.307}]},{"osd":1,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.57199999999999995},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.19600000000000001},"max":{"1min":0.64400000000000002,"5min":1.6859999999999999,"15min":35.128},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.439,"15min":0.56999999999999995},"min":{"1min":0.28199999999999997,"5min":0.26600000000000001,"15min":0.24299999999999999},"max":{"1min":0.54600000000000004,"5min":1.252,"15min":29.631},"last":0.33400000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46200000000000002,"15min":0.60399999999999998},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.23000000000000001},"max":{"1min":0.622,"5min":0.90000000000000002,"15min":35.159999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.61099999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.63900000000000001,"5min":0.88600000000000001,"15min":29.603000000000002},"last":0.432}]},{"osd":4,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63,"15min":0.65900000000000003},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":0.80200000000000005,"5min":0.93899999999999995,"15min":7.3499999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64100000000000001,"15min":0.79300000000000004},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":1.0569999999999999,"15min":35.188000000000002},"last":0.60399999999999998}]},{"osd":5,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.72699999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84999999999999998,"5min":1.165,"15min":23.082999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":0.93899999999999995,"5min":1.323,"15min":7.4020000000000001},"last":0.58599999999999997}]},{"osd":6,"last update":"Sun Sep 15 02:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67300000000000004,"15min":0.78900000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.373},"max":{"1min":0.97699999999999998,"5min":1.365,"15min":35.061999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64600000000000002,"15min":0.74399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44},"max":{"1min":0.90900000000000003,"5min":1.018,"15min":23.129000000000001},"last":0.56799999999999995}]},{"osd":7,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.5,"5min":0.49199999999999999,"15min":0.36699999999999999},"max":{"1min":0.88,"5min":1.1060000000000001,"15min":2.1840000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.80900000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.42099999999999999},"max":{"1min":0.81899999999999995,"5min":0.999,"15min":35.100999999999999},"last":0.55300000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575031,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312044,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059796,"statfs":{"total":96636764160,"available":96317231104,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.253,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":0.55200000000000005,"5min":0.65000000000000002,"15min":1.2370000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":0.57199999999999995,"5min":0.747,"15min":1.1799999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.27100000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59099999999999997,"5min":0.70999999999999996,"15min":1.319},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.73799999999999999,"15min":1.347},"last":0.52700000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.34899999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.55900000000000005,"5min":0.66700000000000004,"15min":1.411},"last":0.502},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.34599999999999997,"5min":0.29199999999999998,"15min":0.252},"max":{"1min":0.622,"5min":0.67500000000000004,"15min":2.9340000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.73099999999999998,"5min":0.95499999999999996,"15min":1.298},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.61699999999999999},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.875,"15min":1.1319999999999999},"last":0.625}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.315},"max":{"1min":0.81999999999999995,"5min":1,"15min":2.9079999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.49199999999999999,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.94999999999999996,"15min":1.367},"last":0.79900000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.008,"15min":1.2050000000000001},"last":0.753},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.96999999999999997,"15min":2.9649999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.50800000000000001,"5min":0.43099999999999999,"15min":0.31900000000000001},"max":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":1.3240000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.44600000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":0.82099999999999995,"5min":0.95199999999999996,"15min":1.167},"last":0.81799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303692,"kb_used_data":6340,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068148,"statfs":{"total":96636764160,"available":96325783552,"internally_reserved":0,"allocated":6492160,"data_stored":3106959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.38900000000000001,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.17899999999999999},"max":{"1min":0.70399999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51800000000000002,"5min":0.65100000000000002,"15min":0.93100000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56000000000000005,"5min":1.405,"15min":1.405},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.441},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.219},"max":{"1min":0.64700000000000002,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.249},"max":{"1min":0.81000000000000005,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.67100000000000004,"5min":2.8490000000000002,"15min":2.8490000000000002},"last":0.50700000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63200000000000001,"15min":0.59999999999999998},"min":{"1min":0.48899999999999999,"5min":0.379,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":1.581,"15min":1.581},"last":0.875},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63800000000000001,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.34899999999999998,"15min":0.29299999999999998},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.95499999999999996},"last":0.65900000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.64200000000000002},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.33300000000000002},"max":{"1min":0.90200000000000002,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.65800000000000003},"min":{"1min":0.56200000000000006,"5min":0.433,"15min":0.371},"max":{"1min":1.004,"5min":1.532,"15min":1.532},"last":0.77100000000000002}]},{"osd":6,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.40400000000000003,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.70299999999999996,"15min":0.66300000000000003},"min":{"1min":0.60899999999999999,"5min":0.46800000000000003,"15min":0.33000000000000002},"max":{"1min":1.0509999999999999,"5min":1.498,"15min":1.498},"last":0.82899999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.53500000000000003,"5min":0.35999999999999999,"15min":0.35699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.32500000000000001},"max":{"1min":0.97399999999999998,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.72999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307920,"kb_used_data":6400,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063920,"statfs":{"total":96636764160,"available":96321454080,"internally_reserved":0,"allocated":6553600,"data_stored":3641449,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.503},"min":{"1min":0.251,"5min":0.24199999999999999,"15min":0.182},"max":{"1min":0.64800000000000002,"5min":0.68500000000000005,"15min":14.630000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46999999999999997,"15min":0.45100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.222},"max":{"1min":0.69299999999999995,"5min":0.85399999999999998,"15min":2.3570000000000002},"last":0.60699999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44500000000000001,"15min":0.47499999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.20200000000000001},"max":{"1min":0.67000000000000004,"5min":0.93899999999999995,"15min":14.679},"last":0.503},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":0.68600000000000005,"15min":1.806},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.21099999999999999},"max":{"1min":0.504,"5min":0.73799999999999999,"15min":2.835},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.48899999999999999,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.26300000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":14.558999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64900000000000002,"15min":0.60499999999999998},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.253},"max":{"1min":0.85599999999999998,"5min":0.98599999999999999,"15min":1.5549999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65300000000000002,"15min":0.61699999999999999},"min":{"1min":0.38900000000000001,"5min":0.33200000000000002,"15min":0.30199999999999999},"max":{"1min":0.79000000000000004,"5min":0.94699999999999995,"15min":1.3120000000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63400000000000001,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.32800000000000001},"max":{"1min":0.878,"5min":0.998,"15min":1.1579999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.34300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":7.2969999999999997},"last":0.67500000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68200000000000005,"15min":0.70599999999999996},"min":{"1min":0.55700000000000005,"5min":0.48599999999999999,"15min":0.35199999999999998},"max":{"1min":0.97499999999999998,"5min":1.0129999999999999,"15min":14.6},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69599999999999995,"15min":0.65400000000000003},"min":{"1min":0.53200000000000003,"5min":0.40999999999999998,"15min":0.35399999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.79600000000000004}]},{"osd":7,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67900000000000005,"15min":0.64000000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.36699999999999999},"max":{"1min":0.95399999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.34300000000000003},"max":{"1min":0.97899999999999998,"5min":1.0029999999999999,"15min":1.093},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:58:19.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:58:19.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.803+0000 7f38d979d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:19.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.803+0000 7f38d979d640 10 monclient: build_initial_monmap 2024-09-15T02:58:19.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.803+0000 7f38d979d640 10 monclient: monmap: 2024-09-15T02:58:19.805 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:19.805 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.805 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:19.804621+0000 2024-09-15T02:58:19.805 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:19.804621+0000 2024-09-15T02:58:19.805 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:19.806 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.806 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:19.806 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:19.806 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:19.806 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:19.806 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:19.806 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.804+0000 7f38d979d640 1 Processor -- start 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.804+0000 7f38d979d640 1 -- start start 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.804+0000 7f38d979d640 10 monclient: init 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient(hunting): picked mon.noname-b con 0x7f38d4140f70 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f38d4141340 0x7f38d413e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient(hunting): picked mon.noname-a con 0x7f38d4141340 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:19.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38d40b0a80 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d413ddd0 con 0x7f38d40b0a80 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d40a6c00 con 0x7f38d4141340 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40a5b30 con 0x7f38d4140f70 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d979d640 10 monclient(hunting): authenticate will time out at 1636.375193s 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f38d4141340 0x7f38d413e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56164/0 (socket says 172.21.15.53:56164) 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 -- 172.21.15.53:0/2654974531 learned_addr learned my addr 172.21.15.53:0/2654974531 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 10 monclient(hunting): get_auth_request con 0x7f38d4141340 auth_method 0 2024-09-15T02:58:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): get_auth_request con 0x7f38d4140f70 auth_method 0 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient(hunting): handle_auth_done global_id 20318 payload 274 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 -- 172.21.15.53:0/2654974531 >> v2:172.21.15.53:3300/0 conn(0x7f38d4141340 msgr2=0x7f38d413e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 --2- 172.21.15.53:0/2654974531 >> v2:172.21.15.53:3300/0 conn(0x7f38d4141340 0x7f38d413e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 -- 172.21.15.53:0/2654974531 >> v1:172.21.15.53:6789/0 conn(0x7f38d40b0a80 legacy=0x7f38d40b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 monclient: found mon.noname-b 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d2ffd640 1 -- 172.21.15.53:0/2654974531 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d413da60 con 0x7f38d4140f70 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.805+0000 7f38d27fc640 1 --2- 172.21.15.53:0/2654974531 >> v2:172.21.15.53:3300/0 conn(0x7f38d4141340 0x7f38d413e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d2ffd640 1 --2- 172.21.15.53:0/2654974531 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40b00f0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009910 tx=0x7f38c40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=149a62b075212f4e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 1 -- 172.21.15.53:0/2654974531 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c402adb0 con 0x7f38d4140f70 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: dump: 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.820 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: _finish_auth 0 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 1 -- 172.21.15.53:0/2654974531 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c4032440 con 0x7f38d4140f70 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d979d640 5 monclient: authenticate success, global_id 20318 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d979d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d979d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 1 -- 172.21.15.53:0/2654974531 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4032820 con 0x7f38d4140f70 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d979d640 4 set_mon_vals no callback set 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.806+0000 7f38d1ffb640 10 monclient: dump: 2024-09-15T02:58:19.821 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 10 monclient: shutdown 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 -- 172.21.15.53:0/2654974531 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 msgr2=0x7f38d40b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 --2- 172.21.15.53:0/2654974531 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40b00f0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009910 tx=0x7f38c40091d0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 -- 172.21.15.53:0/2654974531 shutdown_connections 2024-09-15T02:58:19.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 --2- 172.21.15.53:0/2654974531 >> v2:172.21.15.53:3300/0 conn(0x7f38d4141340 0x7f38d413e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 --2- 172.21.15.53:0/2654974531 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40b00f0 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 -- 172.21.15.53:0/2654974531 >> 172.21.15.53:0/2654974531 conn(0x7f38d4062d90 msgr2=0x7f38d4063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 -- 172.21.15.53:0/2654974531 shutdown_connections 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 1 -- 172.21.15.53:0/2654974531 wait complete. 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 10 monclient: build_initial_monmap 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.807+0000 7f38d979d640 10 monclient: monmap: 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 librados: starting msgr at 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 librados: starting objecter 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 Processor -- start 2024-09-15T02:58:19.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 -- start start 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 librados: setting wanted keys 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 librados: calling monclient init 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient: init 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38d40b0a80 0x7f38d413cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): picked mon.noname-b con 0x7f38d40b0a80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40a8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): picked mon.noname-c con 0x7f38d4140f70 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38d4141340 0x7f38d40a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): picked mon.noname-a con 0x7f38d4141340 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:19.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40ab200 con 0x7f38d4141340 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40ab470 con 0x7f38d40b0a80 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40a92e0 con 0x7f38d4140f70 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d979d640 10 monclient(hunting): authenticate will time out at 1636.378851s 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d27fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40a8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38d4141340 0x7f38d40a9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d27fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40a8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56174/0 (socket says 172.21.15.53:56174) 2024-09-15T02:58:19.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d27fc640 1 -- 172.21.15.53:0/3493150543 learned_addr learned my addr 172.21.15.53:0/3493150543 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d27fc640 10 monclient(hunting): get_auth_request con 0x7f38d4140f70 auth_method 0 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.808+0000 7f38d37fe640 10 monclient(hunting): get_auth_request con 0x7f38d4141340 auth_method 0 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 10 monclient(hunting): handle_auth_done global_id 20321 payload 824 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 1 -- 172.21.15.53:0/3493150543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38d40b0a80 msgr2=0x7f38d413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 1 --2- 172.21.15.53:0/3493150543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38d40b0a80 0x7f38d413cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 1 -- 172.21.15.53:0/3493150543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38d4141340 msgr2=0x7f38d40a9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 1 --2- 172.21.15.53:0/3493150543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38d4141340 0x7f38d40a9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 10 monclient: _finish_hunting 0 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 1 monclient: found mon.noname-c 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 1 -- 172.21.15.53:0/3493150543 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c8009390 con 0x7f38d4140f70 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d27fc640 1 --2- 172.21.15.53:0/3493150543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40a8da0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7f38c800ea70 tx=0x7f38c80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 client.?.objecter ms_handle_connect 0x7f38d4140f70 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 1 -- 172.21.15.53:0/3493150543 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c800ca10 con 0x7f38d4140f70 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: dump: 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.829 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: _finish_auth 0 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 1 -- 172.21.15.53:0/3493150543 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c800cb70 con 0x7f38d4140f70 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 5 monclient: authenticate success, global_id 20321 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 10 monclient: _renew_subs 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c4009450 con 0x7f38d4140f70 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:19.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 1 -- 172.21.15.53:0/3493150543 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8013470 con 0x7f38d4140f70 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38baffd640 10 monclient: dump: 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:19.831 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 10 client.20321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 10 monclient: _renew_subs 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d40a7540 con 0x7f38d4140f70 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d979d640 1 librados: init done 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.809+0000 7f38d17fa640 4 set_mon_vals no callback set 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.810+0000 7f38d979d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.810+0000 7f38d979d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.810+0000 7f38d979d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.810+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d40646e0 con 0x7f38d4140f70 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.811+0000 7f38baffd640 1 -- 172.21.15.53:0/3493150543 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f38c8013820 con 0x7f38d4140f70 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.811+0000 7f38baffd640 1 --2- 172.21.15.53:0/3493150543 >> v2:172.21.15.104:6816/32804 conn(0x7f38c006e800 0x7f38c0070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.811+0000 7f38baffd640 1 -- 172.21.15.53:0/3493150543 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f38c808bcc0 con 0x7f38d4140f70 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.811+0000 7f38baffd640 10 client.20321.objecter ms_dispatch 0x7f38d4000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.811+0000 7f38baffd640 3 client.20321.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.811+0000 7f38baffd640 3 client.20321.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.811+0000 7f38baffd640 20 client.20321.objecter dump_active .. 0 homeless 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.812+0000 7f38d2ffd640 1 --2- 172.21.15.53:0/3493150543 >> v2:172.21.15.104:6816/32804 conn(0x7f38c006e800 0x7f38c0070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.812+0000 7f38d2ffd640 10 monclient: get_auth_request con 0x7f38c006e800 auth_method 0 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.812+0000 7f38d2ffd640 1 --2- 172.21.15.53:0/3493150543 >> v2:172.21.15.104:6816/32804 conn(0x7f38c006e800 0x7f38c0070cb0 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f38c4034000 tx=0x7f38c4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.812+0000 7f38baffd640 10 client.20321.objecter ms_handle_connect 0x7f38c006e800 2024-09-15T02:58:19.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.819+0000 7f38baffd640 1 -- 172.21.15.53:0/3493150543 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f38c8055f60 con 0x7f38d4140f70 2024-09-15T02:58:19.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.819+0000 7f38baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:19.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.819+0000 7f38baffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:19.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.980+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38d40ab5e0 con 0x7f38c006e800 2024-09-15T02:58:19.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.985+0000 7f38baffd640 1 -- 172.21.15.53:0/3493150543 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+153796 (secure 0 0 0) 0x7f38d40ab5e0 con 0x7f38c006e800 2024-09-15T02:58:19.987 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:58:19.988 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:58:19.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 10 librados: watch_flush enter 2024-09-15T02:58:19.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 10 librados: watch_flush exit 2024-09-15T02:58:19.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 20 client.20321.objecter shutdown clearing up homeless session... 2024-09-15T02:58:19.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 10 client.20321.objecter successfully canceled tick 2024-09-15T02:58:19.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 >> v2:172.21.15.104:6816/32804 conn(0x7f38c006e800 msgr2=0x7f38c0070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 --2- 172.21.15.53:0/3493150543 >> v2:172.21.15.104:6816/32804 conn(0x7f38c006e800 0x7f38c0070cb0 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f38c4034000 tx=0x7f38c4032000 comp rx=0 tx=0).stop 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 10 monclient: shutdown 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 msgr2=0x7f38d40a8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 --2- 172.21.15.53:0/3493150543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40a8da0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7f38c800ea70 tx=0x7f38c80090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 shutdown_connections 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 --2- 172.21.15.53:0/3493150543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38d4141340 0x7f38d40a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 --2- 172.21.15.53:0/3493150543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38d4140f70 0x7f38d40a8da0 unknown :-1 s=CLOSED pgs=3129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 --2- 172.21.15.53:0/3493150543 >> v2:172.21.15.104:6816/32804 conn(0x7f38c006e800 0x7f38c0070cb0 unknown :-1 s=CLOSED pgs=3687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 --2- 172.21.15.53:0/3493150543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38d40b0a80 0x7f38d413cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 >> 172.21.15.53:0/3493150543 conn(0x7f38d4062d90 msgr2=0x7f38d40b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 shutdown_connections 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 -- 172.21.15.53:0/3493150543 wait complete. 2024-09-15T02:58:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:19.987+0000 7f38d979d640 1 librados: shutdown 2024-09-15T02:58:19.999 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":1145,"stamp":"2024-09-15T02:58:19.540071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2471344,"kb_used_data":50876,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752503376,"statfs":{"total":773094113280,"available":770563457024,"internally_reserved":0,"allocated":52097024,"data_stored":25969454,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005021"},"pg_stats":[{"pgid":"12.10","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539924+0000","last_change":"2024-09-15T02:44:21.592595+0000","last_active":"2024-09-15T02:58:16.539924+0000","last_peered":"2024-09-15T02:58:16.539924+0000","last_clean":"2024-09-15T02:58:16.539924+0000","last_became_active":"2024-09-15T02:44:21.589224+0000","last_became_peered":"2024-09-15T02:44:21.589224+0000","last_unstale":"2024-09-15T02:58:16.539924+0000","last_undegraded":"2024-09-15T02:58:16.539924+0000","last_fullsized":"2024-09-15T02:58:16.539924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:30:34.422407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836595+0000","last_change":"2024-09-15T02:44:21.472953+0000","last_active":"2024-09-15T02:58:18.836595+0000","last_peered":"2024-09-15T02:58:18.836595+0000","last_clean":"2024-09-15T02:58:18.836595+0000","last_became_active":"2024-09-15T02:44:21.472608+0000","last_became_peered":"2024-09-15T02:44:21.472608+0000","last_unstale":"2024-09-15T02:58:18.836595+0000","last_undegraded":"2024-09-15T02:58:18.836595+0000","last_fullsized":"2024-09-15T02:58:18.836595+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:48.488509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.593941+0000","last_change":"2024-09-15T02:44:21.605637+0000","last_active":"2024-09-15T02:58:13.593941+0000","last_peered":"2024-09-15T02:58:13.593941+0000","last_clean":"2024-09-15T02:58:13.593941+0000","last_became_active":"2024-09-15T02:44:21.604946+0000","last_became_peered":"2024-09-15T02:44:21.604946+0000","last_unstale":"2024-09-15T02:58:13.593941+0000","last_undegraded":"2024-09-15T02:58:13.593941+0000","last_fullsized":"2024-09-15T02:58:13.593941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:21:30.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.538478+0000","last_change":"2024-09-15T02:44:21.631801+0000","last_active":"2024-09-15T02:58:15.538478+0000","last_peered":"2024-09-15T02:58:15.538478+0000","last_clean":"2024-09-15T02:58:15.538478+0000","last_became_active":"2024-09-15T02:44:21.621686+0000","last_became_peered":"2024-09-15T02:44:21.621686+0000","last_unstale":"2024-09-15T02:58:15.538478+0000","last_undegraded":"2024-09-15T02:58:15.538478+0000","last_fullsized":"2024-09-15T02:58:15.538478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:21:17.555939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836680+0000","last_change":"2024-09-15T02:44:21.559633+0000","last_active":"2024-09-15T02:58:18.836680+0000","last_peered":"2024-09-15T02:58:18.836680+0000","last_clean":"2024-09-15T02:58:18.836680+0000","last_became_active":"2024-09-15T02:44:21.558951+0000","last_became_peered":"2024-09-15T02:44:21.558951+0000","last_unstale":"2024-09-15T02:58:18.836680+0000","last_undegraded":"2024-09-15T02:58:18.836680+0000","last_fullsized":"2024-09-15T02:58:18.836680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:27.075112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236232+0000","last_change":"2024-09-15T02:44:21.593138+0000","last_active":"2024-09-15T02:58:17.236232+0000","last_peered":"2024-09-15T02:58:17.236232+0000","last_clean":"2024-09-15T02:58:17.236232+0000","last_became_active":"2024-09-15T02:44:21.592665+0000","last_became_peered":"2024-09-15T02:44:21.592665+0000","last_unstale":"2024-09-15T02:58:17.236232+0000","last_undegraded":"2024-09-15T02:58:17.236232+0000","last_fullsized":"2024-09-15T02:58:17.236232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:20.534135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235717+0000","last_change":"2024-09-15T02:44:21.537510+0000","last_active":"2024-09-15T02:58:17.235717+0000","last_peered":"2024-09-15T02:58:17.235717+0000","last_clean":"2024-09-15T02:58:17.235717+0000","last_became_active":"2024-09-15T02:44:21.537279+0000","last_became_peered":"2024-09-15T02:44:21.537279+0000","last_unstale":"2024-09-15T02:58:17.235717+0000","last_undegraded":"2024-09-15T02:58:17.235717+0000","last_fullsized":"2024-09-15T02:58:17.235717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:55:56.290343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235427+0000","last_change":"2024-09-15T02:44:21.613091+0000","last_active":"2024-09-15T02:58:17.235427+0000","last_peered":"2024-09-15T02:58:17.235427+0000","last_clean":"2024-09-15T02:58:17.235427+0000","last_became_active":"2024-09-15T02:44:21.612841+0000","last_became_peered":"2024-09-15T02:44:21.612841+0000","last_unstale":"2024-09-15T02:58:17.235427+0000","last_undegraded":"2024-09-15T02:58:17.235427+0000","last_fullsized":"2024-09-15T02:58:17.235427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:28:24.283127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537152+0000","last_change":"2024-09-15T02:44:21.543344+0000","last_active":"2024-09-15T02:58:15.537152+0000","last_peered":"2024-09-15T02:58:15.537152+0000","last_clean":"2024-09-15T02:58:15.537152+0000","last_became_active":"2024-09-15T02:44:21.522860+0000","last_became_peered":"2024-09-15T02:44:21.522860+0000","last_unstale":"2024-09-15T02:58:15.537152+0000","last_undegraded":"2024-09-15T02:58:15.537152+0000","last_fullsized":"2024-09-15T02:58:15.537152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:30.830504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541359+0000","last_change":"2024-09-15T02:44:21.538578+0000","last_active":"2024-09-15T02:58:16.541359+0000","last_peered":"2024-09-15T02:58:16.541359+0000","last_clean":"2024-09-15T02:58:16.541359+0000","last_became_active":"2024-09-15T02:44:21.535334+0000","last_became_peered":"2024-09-15T02:44:21.535334+0000","last_unstale":"2024-09-15T02:58:16.541359+0000","last_undegraded":"2024-09-15T02:58:16.541359+0000","last_fullsized":"2024-09-15T02:58:16.541359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:38:25.943125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542872+0000","last_change":"2024-09-15T02:44:21.614405+0000","last_active":"2024-09-15T02:58:17.542872+0000","last_peered":"2024-09-15T02:58:17.542872+0000","last_clean":"2024-09-15T02:58:17.542872+0000","last_became_active":"2024-09-15T02:44:21.614186+0000","last_became_peered":"2024-09-15T02:44:21.614186+0000","last_unstale":"2024-09-15T02:58:17.542872+0000","last_undegraded":"2024-09-15T02:58:17.542872+0000","last_fullsized":"2024-09-15T02:58:17.542872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:52.698525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542738+0000","last_change":"2024-09-15T02:44:21.527859+0000","last_active":"2024-09-15T02:58:17.542738+0000","last_peered":"2024-09-15T02:58:17.542738+0000","last_clean":"2024-09-15T02:58:17.542738+0000","last_became_active":"2024-09-15T02:44:21.527331+0000","last_became_peered":"2024-09-15T02:44:21.527331+0000","last_unstale":"2024-09-15T02:58:17.542738+0000","last_undegraded":"2024-09-15T02:58:17.542738+0000","last_fullsized":"2024-09-15T02:58:17.542738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:55:15.776872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":653,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235679+0000","last_change":"2024-09-15T02:43:23.626724+0000","last_active":"2024-09-15T02:58:17.235679+0000","last_peered":"2024-09-15T02:58:17.235679+0000","last_clean":"2024-09-15T02:58:17.235679+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:58:17.235679+0000","last_undegraded":"2024-09-15T02:58:17.235679+0000","last_fullsized":"2024-09-15T02:58:17.235679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:48.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540373+0000","last_change":"2024-09-15T02:44:21.539798+0000","last_active":"2024-09-15T02:58:17.540373+0000","last_peered":"2024-09-15T02:58:17.540373+0000","last_clean":"2024-09-15T02:58:17.540373+0000","last_became_active":"2024-09-15T02:44:21.530274+0000","last_became_peered":"2024-09-15T02:44:21.530274+0000","last_unstale":"2024-09-15T02:58:17.540373+0000","last_undegraded":"2024-09-15T02:58:17.540373+0000","last_fullsized":"2024-09-15T02:58:17.540373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:24:08.051420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":655,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537819+0000","last_change":"2024-09-15T02:43:23.426048+0000","last_active":"2024-09-15T02:58:15.537819+0000","last_peered":"2024-09-15T02:58:15.537819+0000","last_clean":"2024-09-15T02:58:15.537819+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:58:15.537819+0000","last_undegraded":"2024-09-15T02:58:15.537819+0000","last_fullsized":"2024-09-15T02:58:15.537819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:19.460873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.594062+0000","last_change":"2024-09-15T02:44:21.579868+0000","last_active":"2024-09-15T02:58:13.594062+0000","last_peered":"2024-09-15T02:58:13.594062+0000","last_clean":"2024-09-15T02:58:13.594062+0000","last_became_active":"2024-09-15T02:44:21.579154+0000","last_became_peered":"2024-09-15T02:44:21.579154+0000","last_unstale":"2024-09-15T02:58:13.594062+0000","last_undegraded":"2024-09-15T02:58:13.594062+0000","last_fullsized":"2024-09-15T02:58:13.594062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:45:09.149484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":650,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914260+0000","last_change":"2024-09-15T02:43:23.420483+0000","last_active":"2024-09-15T02:58:16.914260+0000","last_peered":"2024-09-15T02:58:16.914260+0000","last_clean":"2024-09-15T02:58:16.914260+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:58:16.914260+0000","last_undegraded":"2024-09-15T02:58:16.914260+0000","last_fullsized":"2024-09-15T02:58:16.914260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:03:14.233618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.594285+0000","last_change":"2024-09-15T02:44:21.537181+0000","last_active":"2024-09-15T02:58:13.594285+0000","last_peered":"2024-09-15T02:58:13.594285+0000","last_clean":"2024-09-15T02:58:13.594285+0000","last_became_active":"2024-09-15T02:44:21.533501+0000","last_became_peered":"2024-09-15T02:44:21.533501+0000","last_unstale":"2024-09-15T02:58:13.594285+0000","last_undegraded":"2024-09-15T02:58:13.594285+0000","last_fullsized":"2024-09-15T02:58:13.594285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:42.258617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":655,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537074+0000","last_change":"2024-09-15T02:43:23.426071+0000","last_active":"2024-09-15T02:58:15.537074+0000","last_peered":"2024-09-15T02:58:15.537074+0000","last_clean":"2024-09-15T02:58:15.537074+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:58:15.537074+0000","last_undegraded":"2024-09-15T02:58:15.537074+0000","last_fullsized":"2024-09-15T02:58:15.537074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:00:03.954753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540360+0000","last_change":"2024-09-15T02:44:21.561913+0000","last_active":"2024-09-15T02:58:17.540360+0000","last_peered":"2024-09-15T02:58:17.540360+0000","last_clean":"2024-09-15T02:58:17.540360+0000","last_became_active":"2024-09-15T02:44:21.560425+0000","last_became_peered":"2024-09-15T02:44:21.560425+0000","last_unstale":"2024-09-15T02:58:17.540360+0000","last_undegraded":"2024-09-15T02:58:17.540360+0000","last_fullsized":"2024-09-15T02:58:17.540360+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:40:57.301643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":658,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836590+0000","last_change":"2024-09-15T02:43:23.423983+0000","last_active":"2024-09-15T02:58:18.836590+0000","last_peered":"2024-09-15T02:58:18.836590+0000","last_clean":"2024-09-15T02:58:18.836590+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:58:18.836590+0000","last_undegraded":"2024-09-15T02:58:18.836590+0000","last_fullsized":"2024-09-15T02:58:18.836590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:45.683030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539798+0000","last_change":"2024-09-15T02:44:21.500932+0000","last_active":"2024-09-15T02:58:16.539798+0000","last_peered":"2024-09-15T02:58:16.539798+0000","last_clean":"2024-09-15T02:58:16.539798+0000","last_became_active":"2024-09-15T02:44:21.500646+0000","last_became_peered":"2024-09-15T02:44:21.500646+0000","last_unstale":"2024-09-15T02:58:16.539798+0000","last_undegraded":"2024-09-15T02:58:16.539798+0000","last_fullsized":"2024-09-15T02:58:16.539798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:08.733289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":663,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540358+0000","last_change":"2024-09-15T02:43:23.419938+0000","last_active":"2024-09-15T02:58:17.540358+0000","last_peered":"2024-09-15T02:58:17.540358+0000","last_clean":"2024-09-15T02:58:17.540358+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:58:17.540358+0000","last_undegraded":"2024-09-15T02:58:17.540358+0000","last_fullsized":"2024-09-15T02:58:17.540358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:09:35.753407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235268+0000","last_change":"2024-09-15T02:44:21.504909+0000","last_active":"2024-09-15T02:58:17.235268+0000","last_peered":"2024-09-15T02:58:17.235268+0000","last_clean":"2024-09-15T02:58:17.235268+0000","last_became_active":"2024-09-15T02:44:21.504665+0000","last_became_peered":"2024-09-15T02:44:21.504665+0000","last_unstale":"2024-09-15T02:58:17.235268+0000","last_undegraded":"2024-09-15T02:58:17.235268+0000","last_fullsized":"2024-09-15T02:58:17.235268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:17:58.326032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":649,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913924+0000","last_change":"2024-09-15T02:43:23.420430+0000","last_active":"2024-09-15T02:58:16.913924+0000","last_peered":"2024-09-15T02:58:16.913924+0000","last_clean":"2024-09-15T02:58:16.913924+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:58:16.913924+0000","last_undegraded":"2024-09-15T02:58:16.913924+0000","last_fullsized":"2024-09-15T02:58:16.913924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:05:17.337850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.543353+0000","last_change":"2024-09-15T02:44:21.645181+0000","last_active":"2024-09-15T02:58:16.543353+0000","last_peered":"2024-09-15T02:58:16.543353+0000","last_clean":"2024-09-15T02:58:16.543353+0000","last_became_active":"2024-09-15T02:44:21.644939+0000","last_became_peered":"2024-09-15T02:44:21.644939+0000","last_unstale":"2024-09-15T02:58:16.543353+0000","last_undegraded":"2024-09-15T02:58:16.543353+0000","last_fullsized":"2024-09-15T02:58:16.543353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:22:20.773021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":657,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836538+0000","last_change":"2024-09-15T02:43:23.423477+0000","last_active":"2024-09-15T02:58:18.836538+0000","last_peered":"2024-09-15T02:58:18.836538+0000","last_clean":"2024-09-15T02:58:18.836538+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:58:18.836538+0000","last_undegraded":"2024-09-15T02:58:18.836538+0000","last_fullsized":"2024-09-15T02:58:18.836538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:05:30.605263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":735,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913790+0000","last_change":"2024-09-15T02:43:20.387731+0000","last_active":"2024-09-15T02:58:16.913790+0000","last_peered":"2024-09-15T02:58:16.913790+0000","last_clean":"2024-09-15T02:58:16.913790+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:58:16.913790+0000","last_undegraded":"2024-09-15T02:58:16.913790+0000","last_fullsized":"2024-09-15T02:58:16.913790+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_clean_scrub_stamp":"2024-09-15T02:43:19.374225+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:33:29.816927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":639,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914166+0000","last_change":"2024-09-15T02:44:21.629491+0000","last_active":"2024-09-15T02:58:16.914166+0000","last_peered":"2024-09-15T02:58:16.914166+0000","last_clean":"2024-09-15T02:58:16.914166+0000","last_became_active":"2024-09-15T02:44:21.629239+0000","last_became_peered":"2024-09-15T02:44:21.629239+0000","last_unstale":"2024-09-15T02:58:16.914166+0000","last_undegraded":"2024-09-15T02:58:16.914166+0000","last_fullsized":"2024-09-15T02:58:16.914166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:56.354264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.538060+0000","last_change":"2024-09-15T02:44:21.573593+0000","last_active":"2024-09-15T02:58:15.538060+0000","last_peered":"2024-09-15T02:58:15.538060+0000","last_clean":"2024-09-15T02:58:15.538060+0000","last_became_active":"2024-09-15T02:44:21.571415+0000","last_became_peered":"2024-09-15T02:44:21.571415+0000","last_unstale":"2024-09-15T02:58:15.538060+0000","last_undegraded":"2024-09-15T02:58:15.538060+0000","last_fullsized":"2024-09-15T02:58:15.538060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:12.880270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537934+0000","last_change":"2024-09-15T02:44:21.629604+0000","last_active":"2024-09-15T02:58:15.537934+0000","last_peered":"2024-09-15T02:58:15.537934+0000","last_clean":"2024-09-15T02:58:15.537934+0000","last_became_active":"2024-09-15T02:44:21.628817+0000","last_became_peered":"2024-09-15T02:44:21.628817+0000","last_unstale":"2024-09-15T02:58:15.537934+0000","last_undegraded":"2024-09-15T02:58:15.537934+0000","last_fullsized":"2024-09-15T02:58:15.537934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:52:41.637023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236078+0000","last_change":"2024-09-15T02:44:21.583579+0000","last_active":"2024-09-15T02:58:17.236078+0000","last_peered":"2024-09-15T02:58:17.236078+0000","last_clean":"2024-09-15T02:58:17.236078+0000","last_became_active":"2024-09-15T02:44:21.583340+0000","last_became_peered":"2024-09-15T02:44:21.583340+0000","last_unstale":"2024-09-15T02:58:17.236078+0000","last_undegraded":"2024-09-15T02:58:17.236078+0000","last_fullsized":"2024-09-15T02:58:17.236078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:01:58.119370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235646+0000","last_change":"2024-09-15T02:44:21.616190+0000","last_active":"2024-09-15T02:58:17.235646+0000","last_peered":"2024-09-15T02:58:17.235646+0000","last_clean":"2024-09-15T02:58:17.235646+0000","last_became_active":"2024-09-15T02:44:21.615991+0000","last_became_peered":"2024-09-15T02:44:21.615991+0000","last_unstale":"2024-09-15T02:58:17.235646+0000","last_undegraded":"2024-09-15T02:58:17.235646+0000","last_fullsized":"2024-09-15T02:58:17.235646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:25:27.900125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539926+0000","last_change":"2024-09-15T02:44:21.608529+0000","last_active":"2024-09-15T02:58:16.539926+0000","last_peered":"2024-09-15T02:58:16.539926+0000","last_clean":"2024-09-15T02:58:16.539926+0000","last_became_active":"2024-09-15T02:44:21.588781+0000","last_became_peered":"2024-09-15T02:44:21.588781+0000","last_unstale":"2024-09-15T02:58:16.539926+0000","last_undegraded":"2024-09-15T02:58:16.539926+0000","last_fullsized":"2024-09-15T02:58:16.539926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:49.754802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.614367+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.613899+0000","last_became_peered":"2024-09-15T02:44:21.613899+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:18:47.868539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1","version":"22'7","reported_seq":655,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235332+0000","last_change":"2024-09-15T02:44:21.595390+0000","last_active":"2024-09-15T02:58:17.235332+0000","last_peered":"2024-09-15T02:58:17.235332+0000","last_clean":"2024-09-15T02:58:17.235332+0000","last_became_active":"2024-09-15T02:44:21.594670+0000","last_became_peered":"2024-09-15T02:44:21.594670+0000","last_unstale":"2024-09-15T02:58:17.235332+0000","last_undegraded":"2024-09-15T02:58:17.235332+0000","last_fullsized":"2024-09-15T02:58:17.235332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:52:07.118143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.536146+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.535920+0000","last_became_peered":"2024-09-15T02:44:21.535920+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.095882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235267+0000","last_change":"2024-09-15T02:44:21.578890+0000","last_active":"2024-09-15T02:58:17.235267+0000","last_peered":"2024-09-15T02:58:17.235267+0000","last_clean":"2024-09-15T02:58:17.235267+0000","last_became_active":"2024-09-15T02:44:21.578571+0000","last_became_peered":"2024-09-15T02:44:21.578571+0000","last_unstale":"2024-09-15T02:58:17.235267+0000","last_undegraded":"2024-09-15T02:58:17.235267+0000","last_fullsized":"2024-09-15T02:58:17.235267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:01.395702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542928+0000","last_change":"2024-09-15T02:44:21.556333+0000","last_active":"2024-09-15T02:58:17.542928+0000","last_peered":"2024-09-15T02:58:17.542928+0000","last_clean":"2024-09-15T02:58:17.542928+0000","last_became_active":"2024-09-15T02:44:21.556078+0000","last_became_peered":"2024-09-15T02:44:21.556078+0000","last_unstale":"2024-09-15T02:58:17.542928+0000","last_undegraded":"2024-09-15T02:58:17.542928+0000","last_fullsized":"2024-09-15T02:58:17.542928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:35:40.520618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541145+0000","last_change":"2024-09-15T02:44:21.583194+0000","last_active":"2024-09-15T02:58:16.541145+0000","last_peered":"2024-09-15T02:58:16.541145+0000","last_clean":"2024-09-15T02:58:16.541145+0000","last_became_active":"2024-09-15T02:44:21.582718+0000","last_became_peered":"2024-09-15T02:44:21.582718+0000","last_unstale":"2024-09-15T02:58:16.541145+0000","last_undegraded":"2024-09-15T02:58:16.541145+0000","last_fullsized":"2024-09-15T02:58:16.541145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:48:37.528157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.540867+0000","last_change":"2024-09-15T02:44:21.584252+0000","last_active":"2024-09-15T02:58:16.540867+0000","last_peered":"2024-09-15T02:58:16.540867+0000","last_clean":"2024-09-15T02:58:16.540867+0000","last_became_active":"2024-09-15T02:44:21.584028+0000","last_became_peered":"2024-09-15T02:44:21.584028+0000","last_unstale":"2024-09-15T02:58:16.540867+0000","last_undegraded":"2024-09-15T02:58:16.540867+0000","last_fullsized":"2024-09-15T02:58:16.540867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:14.221941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575031,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312068,"kb_used_data":6388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059772,"statfs":{"total":96636764160,"available":96317206528,"internally_reserved":0,"allocated":6541312,"data_stored":3635868,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.69399999999999995,"5min":0.75,"15min":1.3959999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.28100000000000003},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.80500000000000005},"last":0.76500000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.55300000000000005},"min":{"1min":0.46100000000000002,"5min":0.33600000000000002,"15min":0.23400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.94599999999999995},"last":0.745},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59799999999999998,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.27300000000000002},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":1.4430000000000001},"last":0.67900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60799999999999998,"15min":0.56200000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.27000000000000002},"max":{"1min":0.745,"5min":0.78600000000000003,"15min":0.95899999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.20300000000000001},"max":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":1.478},"last":0.80300000000000005}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57599999999999996,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.28299999999999997},"max":{"1min":0.72699999999999998,"5min":0.83699999999999997,"15min":2.4300000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.57299999999999995},"min":{"1min":0.502,"5min":0.40600000000000003,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.89700000000000002,"15min":1.6120000000000001},"last":0.71599999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45100000000000001,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.24199999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.755},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.28000000000000003},"max":{"1min":0.70599999999999996,"5min":0.83399999999999996,"15min":0.95899999999999996},"last":0.626}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":0.45400000000000001},"min":{"1min":0.35499999999999998,"5min":0.30099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72899999999999998,"15min":0.88900000000000001},"last":0.502},{"interface":"front","average":{"1min":0.44,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":2.2400000000000002},"last":0.57699999999999996}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.77600000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.52500000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.79300000000000004,"5min":0.89700000000000002,"15min":0.92800000000000005},"last":0.55000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312012,"kb_used_data":6340,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059828,"statfs":{"total":96636764160,"available":96317263872,"internally_reserved":0,"allocated":6492160,"data_stored":3106945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.34599999999999997,"15min":0.27000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":5.0250000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.874},"min":{"1min":0.46600000000000003,"5min":0.37,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":0.85699999999999998,"15min":71.516999999999996},"last":0.45400000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.94699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.25900000000000001},"max":{"1min":0.69399999999999995,"5min":0.84199999999999997,"15min":93.381},"last":0.5},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56100000000000005,"15min":0.86399999999999999},"min":{"1min":0.39300000000000002,"5min":0.33500000000000002,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.90900000000000003,"15min":79.905000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.66400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.93899999999999995,"15min":29.305},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.318},"max":{"1min":0.66500000000000004,"5min":0.85499999999999998,"15min":9.8510000000000009},"last":0.54800000000000004}]},{"osd":3,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.69499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.255},"max":{"1min":0.75900000000000001,"5min":0.96599999999999997,"15min":34.877000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.29299999999999998},"max":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":31.657},"last":0.51000000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.51000000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.20300000000000001},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":20.212},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.442,"15min":0.46300000000000002},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.64700000000000002,"5min":0.71599999999999997,"15min":5.6909999999999998},"last":0.35999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.215},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80000000000000004},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.248},"max":{"1min":0.71499999999999997,"5min":0.81799999999999995,"15min":3.012},"last":0.58699999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47099999999999997,"15min":0.59499999999999997},"min":{"1min":0.35099999999999998,"5min":0.32500000000000001,"15min":0.24299999999999999},"max":{"1min":0.68999999999999995,"5min":0.71299999999999997,"15min":28.756},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.48899999999999999,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.32000000000000001,"15min":0.249},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":27.602},"last":0.436}]}]},{"osd":5,"up_from":13,"seq":55834575032,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":6352,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":6504448,"data_stored":3112540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.69899999999999995},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":17.704999999999998},"last":0.435},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.621,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.78100000000000003,"5min":0.872,"15min":17.465},"last":0.54900000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.22900000000000001},"max":{"1min":0.81999999999999995,"5min":0.83999999999999997,"15min":7.1710000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.42999999999999999,"5min":0.33300000000000002,"15min":0.27700000000000002},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":6.7999999999999998},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1000000000000001,"15min":8.2789999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.60299999999999998},"min":{"1min":0.51600000000000001,"5min":0.33800000000000002,"15min":0.29099999999999998},"max":{"1min":0.79300000000000004,"5min":1.133,"15min":3.0059999999999998},"last":0.61299999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.628},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.25700000000000001},"max":{"1min":0.81299999999999994,"5min":0.872,"15min":7.8090000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63600000000000001,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.38200000000000001,"15min":0.28599999999999998},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":8.6829999999999998},"last":0.72399999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46300000000000002,"15min":0.44600000000000001},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.216},"max":{"1min":0.68100000000000005,"5min":0.82099999999999995,"15min":0.98699999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.23000000000000001},"max":{"1min":0.68300000000000005,"5min":0.78000000000000003,"15min":2.0750000000000002},"last":0.63200000000000001}]},{"osd":6,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.22800000000000001},"max":{"1min":0.67000000000000004,"5min":0.81799999999999995,"15min":3.4420000000000002},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.39100000000000001,"5min":0.33300000000000002,"15min":0.25},"max":{"1min":0.68899999999999995,"5min":0.86299999999999999,"15min":3.0510000000000002},"last":0.65200000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.253},"max":{"1min":0.69699999999999995,"5min":0.79400000000000004,"15min":2.9870000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.94399999999999995,"15min":1.337},"last":0.78200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307884,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063956,"statfs":{"total":96636764160,"available":96321490944,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.86299999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.316},"max":{"1min":0.72099999999999997,"5min":0.873,"15min":0.93100000000000005},"last":0.72699999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.69699999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.32800000000000001},"max":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.379,"15min":0.223},"max":{"1min":0.73699999999999999,"5min":0.89400000000000002,"15min":0.94199999999999995},"last":0.754},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.83499999999999996}]},{"osd":3,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.47799999999999998,"5min":0.36699999999999999,"15min":0.26800000000000002},"max":{"1min":0.82499999999999996,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60399999999999998,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.36099999999999999},"max":{"1min":0.752,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.23200000000000001},"max":{"1min":0.67400000000000004,"5min":0.69699999999999995,"15min":0.81599999999999995},"last":0.47999999999999998}]},{"osd":6,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.497},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42099999999999999,"15min":0.44700000000000001},"min":{"1min":0.316,"5min":0.31,"15min":0.251},"max":{"1min":0.505,"5min":0.65700000000000003,"15min":0.69699999999999995},"last":0.59599999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.78600000000000003,"5min":1.48,"15min":1.48},"last":0.439},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.49199999999999999,"15min":0.50600000000000001},"min":{"1min":0.373,"5min":0.31900000000000001,"15min":0.28000000000000003},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.81799999999999995},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575031,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307852,"kb_used_data":6336,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063988,"statfs":{"total":96636764160,"available":96321523712,"internally_reserved":0,"allocated":6488064,"data_stored":3127885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.371,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.52200000000000002,"5min":0.72399999999999998,"15min":29.562999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47899999999999998,"15min":0.44400000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":0.88600000000000001,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":0.307}]},{"osd":1,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.57199999999999995},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.19600000000000001},"max":{"1min":0.64400000000000002,"5min":1.6859999999999999,"15min":35.128},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.439,"15min":0.56999999999999995},"min":{"1min":0.28199999999999997,"5min":0.26600000000000001,"15min":0.24299999999999999},"max":{"1min":0.54600000000000004,"5min":1.252,"15min":29.631},"last":0.33400000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46200000000000002,"15min":0.60399999999999998},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.23000000000000001},"max":{"1min":0.622,"5min":0.90000000000000002,"15min":35.159999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.61099999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.63900000000000001,"5min":0.88600000000000001,"15min":29.603000000000002},"last":0.432}]},{"osd":4,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63,"15min":0.65900000000000003},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":0.80200000000000005,"5min":0.93899999999999995,"15min":7.3499999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64100000000000001,"15min":0.79300000000000004},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":1.0569999999999999,"15min":35.188000000000002},"last":0.60399999999999998}]},{"osd":5,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.72699999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84999999999999998,"5min":1.165,"15min":23.082999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":0.93899999999999995,"5min":1.323,"15min":7.4020000000000001},"last":0.58599999999999997}]},{"osd":6,"last update":"Sun Sep 15 02:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67300000000000004,"15min":0.78900000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.373},"max":{"1min":0.97699999999999998,"5min":1.365,"15min":35.061999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64600000000000002,"15min":0.74399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44},"max":{"1min":0.90900000000000003,"5min":1.018,"15min":23.129000000000001},"last":0.56799999999999995}]},{"osd":7,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.5,"5min":0.49199999999999999,"15min":0.36699999999999999},"max":{"1min":0.88,"5min":1.1060000000000001,"15min":2.1840000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.80900000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.42099999999999999},"max":{"1min":0.81899999999999995,"5min":0.999,"15min":35.100999999999999},"last":0.55300000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575031,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312044,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059796,"statfs":{"total":96636764160,"available":96317231104,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.253,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":0.55200000000000005,"5min":0.65000000000000002,"15min":1.2370000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":0.57199999999999995,"5min":0.747,"15min":1.1799999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.27100000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59099999999999997,"5min":0.70999999999999996,"15min":1.319},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.73799999999999999,"15min":1.347},"last":0.52700000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.34899999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.55900000000000005,"5min":0.66700000000000004,"15min":1.411},"last":0.502},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.34599999999999997,"5min":0.29199999999999998,"15min":0.252},"max":{"1min":0.622,"5min":0.67500000000000004,"15min":2.9340000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.73099999999999998,"5min":0.95499999999999996,"15min":1.298},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.61699999999999999},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.875,"15min":1.1319999999999999},"last":0.625}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.315},"max":{"1min":0.81999999999999995,"5min":1,"15min":2.9079999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.49199999999999999,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.94999999999999996,"15min":1.367},"last":0.79900000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.008,"15min":1.2050000000000001},"last":0.753},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.96999999999999997,"15min":2.9649999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.50800000000000001,"5min":0.43099999999999999,"15min":0.31900000000000001},"max":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":1.3240000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.44600000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":0.82099999999999995,"5min":0.95199999999999996,"15min":1.167},"last":0.81799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303692,"kb_used_data":6340,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068148,"statfs":{"total":96636764160,"available":96325783552,"internally_reserved":0,"allocated":6492160,"data_stored":3106959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.38900000000000001,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.17899999999999999},"max":{"1min":0.70399999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51800000000000002,"5min":0.65100000000000002,"15min":0.93100000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56000000000000005,"5min":1.405,"15min":1.405},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.441},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.219},"max":{"1min":0.64700000000000002,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.249},"max":{"1min":0.81000000000000005,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.67100000000000004,"5min":2.8490000000000002,"15min":2.8490000000000002},"last":0.50700000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63200000000000001,"15min":0.59999999999999998},"min":{"1min":0.48899999999999999,"5min":0.379,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":1.581,"15min":1.581},"last":0.875},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63800000000000001,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.34899999999999998,"15min":0.29299999999999998},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.95499999999999996},"last":0.65900000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.64200000000000002},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.33300000000000002},"max":{"1min":0.90200000000000002,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.65800000000000003},"min":{"1min":0.56200000000000006,"5min":0.433,"15min":0.371},"max":{"1min":1.004,"5min":1.532,"15min":1.532},"last":0.77100000000000002}]},{"osd":6,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.40400000000000003,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.70299999999999996,"15min":0.66300000000000003},"min":{"1min":0.60899999999999999,"5min":0.46800000000000003,"15min":0.33000000000000002},"max":{"1min":1.0509999999999999,"5min":1.498,"15min":1.498},"last":0.82899999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.53500000000000003,"5min":0.35999999999999999,"15min":0.35699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.32500000000000001},"max":{"1min":0.97399999999999998,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.72999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307920,"kb_used_data":6400,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063920,"statfs":{"total":96636764160,"available":96321454080,"internally_reserved":0,"allocated":6553600,"data_stored":3641449,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.503},"min":{"1min":0.251,"5min":0.24199999999999999,"15min":0.182},"max":{"1min":0.64800000000000002,"5min":0.68500000000000005,"15min":14.630000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46999999999999997,"15min":0.45100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.222},"max":{"1min":0.69299999999999995,"5min":0.85399999999999998,"15min":2.3570000000000002},"last":0.60699999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44500000000000001,"15min":0.47499999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.20200000000000001},"max":{"1min":0.67000000000000004,"5min":0.93899999999999995,"15min":14.679},"last":0.503},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":0.68600000000000005,"15min":1.806},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.21099999999999999},"max":{"1min":0.504,"5min":0.73799999999999999,"15min":2.835},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.48899999999999999,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.26300000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":14.558999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64900000000000002,"15min":0.60499999999999998},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.253},"max":{"1min":0.85599999999999998,"5min":0.98599999999999999,"15min":1.5549999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65300000000000002,"15min":0.61699999999999999},"min":{"1min":0.38900000000000001,"5min":0.33200000000000002,"15min":0.30199999999999999},"max":{"1min":0.79000000000000004,"5min":0.94699999999999995,"15min":1.3120000000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63400000000000001,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.32800000000000001},"max":{"1min":0.878,"5min":0.998,"15min":1.1579999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.34300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":7.2969999999999997},"last":0.67500000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68200000000000005,"15min":0.70599999999999996},"min":{"1min":0.55700000000000005,"5min":0.48599999999999999,"15min":0.35199999999999998},"max":{"1min":0.97499999999999998,"5min":1.0129999999999999,"15min":14.6},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69599999999999995,"15min":0.65400000000000003},"min":{"1min":0.53200000000000003,"5min":0.40999999999999998,"15min":0.35399999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.79600000000000004}]},{"osd":7,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67900000000000005,"15min":0.64000000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.36699999999999999},"max":{"1min":0.95399999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.34300000000000003},"max":{"1min":0.97899999999999998,"5min":1.0029999999999999,"15min":1.093},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:58:20.002 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T02:58:20.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.098+0000 7f65d375d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.098+0000 7f65d375d640 10 monclient: build_initial_monmap 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.098+0000 7f65d375d640 10 monclient: monmap: 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:20.098951+0000 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:20.098951+0000 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:20.100 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.098+0000 7f65d375d640 1 Processor -- start 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 1 -- start start 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient: init 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65cc0b0e80 0x7f65cc0b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient(hunting): picked mon.noname-a con 0x7f65cc0b0e80 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65cc0b1250 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:20.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient(hunting): picked mon.noname-b con 0x7f65cc13e600 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc13da60 con 0x7f65cc0b1250 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc0b0990 con 0x7f65cc0b0e80 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0a6a30 con 0x7f65cc13e600 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d375d640 10 monclient(hunting): authenticate will time out at 1636.669749s 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65cc0b0e80 0x7f65cc0b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65cc0b0e80 0x7f65cc0b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52326/0 (socket says 172.21.15.53:52326) 2024-09-15T02:58:20.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d1cfa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 1 -- 172.21.15.53:0/3198919748 learned_addr learned my addr 172.21.15.53:0/3198919748 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 10 monclient(hunting): get_auth_request con 0x7f65cc0b0e80 auth_method 0 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d1cfa640 10 monclient(hunting): get_auth_request con 0x7f65cc13e600 auth_method 0 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d1cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d1cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d1cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d14f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.099+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_done global_id 20324 payload 274 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 1 -- 172.21.15.53:0/3198919748 >> v2:172.21.15.53:3300/0 conn(0x7f65cc0b0e80 msgr2=0x7f65cc0b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 1 --2- 172.21.15.53:0/3198919748 >> v2:172.21.15.53:3300/0 conn(0x7f65cc0b0e80 0x7f65cc0b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 1 -- 172.21.15.53:0/3198919748 >> v1:172.21.15.104:6789/0 conn(0x7f65cc0b1250 legacy=0x7f65cc13def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 10 monclient: _finish_hunting 0 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 1 monclient: found mon.noname-b 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 1 -- 172.21.15.53:0/3198919748 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc0a5a00 con 0x7f65cc13e600 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d14f9640 1 --2- 172.21.15.53:0/3198919748 >> v2:172.21.15.53:3300/0 conn(0x7f65cc0b0e80 0x7f65cc0b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.100+0000 7f65d1cfa640 1 --2- 172.21.15.53:0/3198919748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0af3c0 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009e60 tx=0x7f65c4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73d05856176f49e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 1 -- 172.21.15.53:0/3198919748 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4007550 con 0x7f65cc13e600 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: dump: 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.112 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: _finish_auth 0 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 5 monclient: authenticate success, global_id 20324 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 1 -- 172.21.15.53:0/3198919748 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c402d050 con 0x7f65cc13e600 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 4 set_mon_vals no callback set 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 1 -- 172.21.15.53:0/3198919748 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4007920 con 0x7f65cc13e600 2024-09-15T02:58:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65c3fff640 10 monclient: dump: 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.114 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 10 monclient: shutdown 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 -- 172.21.15.53:0/3198919748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 msgr2=0x7f65cc0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 --2- 172.21.15.53:0/3198919748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0af3c0 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009e60 tx=0x7f65c4009780 comp rx=0 tx=0).stop 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 -- 172.21.15.53:0/3198919748 shutdown_connections 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 --2- 172.21.15.53:0/3198919748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0af3c0 unknown :-1 s=CLOSED pgs=3130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 --2- 172.21.15.53:0/3198919748 >> v2:172.21.15.53:3300/0 conn(0x7f65cc0b0e80 0x7f65cc0b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 -- 172.21.15.53:0/3198919748 >> 172.21.15.53:0/3198919748 conn(0x7f65cc062d90 msgr2=0x7f65cc063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 -- 172.21.15.53:0/3198919748 shutdown_connections 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.101+0000 7f65d375d640 1 -- 172.21.15.53:0/3198919748 wait complete. 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 10 monclient: build_initial_monmap 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 10 monclient: monmap: 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 1 librados: starting msgr at 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 1 librados: starting objecter 2024-09-15T02:58:20.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 1 Processor -- start 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 1 -- start start 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 1 librados: setting wanted keys 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 1 librados: calling monclient init 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.102+0000 7f65d375d640 10 monclient: init 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65cc0b0e80 0x7f65cc0a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): picked mon.noname-b con 0x7f65cc0b0e80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0b1250 0x7f65cc0a7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): picked mon.noname-a con 0x7f65cc0b1250 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): picked mon.noname-c con 0x7f65cc13e600 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0abfe0 con 0x7f65cc0b1250 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0a73e0 con 0x7f65cc0b0e80 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0aa360 con 0x7f65cc13e600 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d375d640 10 monclient(hunting): authenticate will time out at 1636.673365s 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0b1250 0x7f65cc0a7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0a7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56206/0 (socket says 172.21.15.53:56206) 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0b1250 0x7f65cc0a7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52342/0 (socket says 172.21.15.53:52342) 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 -- 172.21.15.53:0/1185273438 learned_addr learned my addr 172.21.15.53:0/1185273438 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): get_auth_request con 0x7f65cc13e600 auth_method 0 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 10 monclient(hunting): get_auth_request con 0x7f65cc0b1250 auth_method 0 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65cc0b0e80 0x7f65cc0a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 10 monclient(hunting): get_auth_request con 0x7f65cc0b0e80 auth_method 0 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d14f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient(hunting): handle_auth_done global_id 20327 payload 824 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 -- 172.21.15.53:0/1185273438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65cc0b0e80 msgr2=0x7f65cc0a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65cc0b0e80 0x7f65cc0a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 -- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0b1250 msgr2=0x7f65cc0a7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0b1250 0x7f65cc0a7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient: _finish_hunting 0 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 monclient: found mon.noname-c 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d1cfa640 1 -- 172.21.15.53:0/1185273438 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c8009450 con 0x7f65cc13e600 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.103+0000 7f65d0cf8640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0b1250 0x7f65cc0a7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d1cfa640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0a7e40 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f65c40099f0 tx=0x7f65c4037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 client.?.objecter ms_handle_connect 0x7f65cc13e600 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 1 -- 172.21.15.53:0/1185273438 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4031960 con 0x7f65cc13e600 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: dump: 2024-09-15T02:58:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: _finish_auth 0 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 5 monclient: authenticate success, global_id 20327 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 1 -- 172.21.15.53:0/1185273438 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4040070 con 0x7f65cc13e600 2024-09-15T02:58:20.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 1 -- 172.21.15.53:0/1185273438 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4031d10 con 0x7f65cc13e600 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 10 monclient: _renew_subs 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c4009a20 con 0x7f65cc13e600 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c17fa640 10 monclient: dump: 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.123 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65c37fe640 4 set_mon_vals no callback set 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 10 client.20327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 10 monclient: _renew_subs 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc0a85a0 con 0x7f65cc13e600 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.104+0000 7f65d375d640 1 librados: init done 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.105+0000 7f65d375d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.105+0000 7f65d375d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.105+0000 7f65d375d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.105+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc063ff0 con 0x7f65cc13e600 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.105+0000 7f65c17fa640 1 -- 172.21.15.53:0/1185273438 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f65c4042030 con 0x7f65cc13e600 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65c17fa640 1 --2- 172.21.15.53:0/1185273438 >> v2:172.21.15.104:6816/32804 conn(0x7f65b006e800 0x7f65b0070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65c17fa640 1 -- 172.21.15.53:0/1185273438 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f65c40ad2f0 con 0x7f65cc13e600 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65c17fa640 10 client.20327.objecter ms_dispatch 0x7f65cc000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65c17fa640 3 client.20327.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65c17fa640 3 client.20327.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65d14f9640 1 --2- 172.21.15.53:0/1185273438 >> v2:172.21.15.104:6816/32804 conn(0x7f65b006e800 0x7f65b0070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65c17fa640 20 client.20327.objecter dump_active .. 0 homeless 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65d14f9640 10 monclient: get_auth_request con 0x7f65b006e800 auth_method 0 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65c17fa640 10 client.20327.objecter ms_handle_connect 0x7f65b006e800 2024-09-15T02:58:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.106+0000 7f65d14f9640 1 --2- 172.21.15.53:0/1185273438 >> v2:172.21.15.104:6816/32804 conn(0x7f65b006e800 0x7f65b0070cb0 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7f65c8006c90 tx=0x7f65c80091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:20.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.110+0000 7f65c17fa640 1 -- 172.21.15.53:0/1185273438 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f65c4077590 con 0x7f65cc13e600 2024-09-15T02:58:20.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.110+0000 7f65c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:20.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.110+0000 7f65c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:20.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.269+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65cc0ac150 con 0x7f65b006e800 2024-09-15T02:58:20.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.274+0000 7f65c17fa640 1 -- 172.21.15.53:0/1185273438 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+153796 (secure 0 0 0) 0x7f65cc0ac150 con 0x7f65b006e800 2024-09-15T02:58:20.277 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:58:20.278 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 10 librados: watch_flush enter 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 10 librados: watch_flush exit 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 20 client.20327.objecter shutdown clearing up homeless session... 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 10 client.20327.objecter successfully canceled tick 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 >> v2:172.21.15.104:6816/32804 conn(0x7f65b006e800 msgr2=0x7f65b0070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 --2- 172.21.15.53:0/1185273438 >> v2:172.21.15.104:6816/32804 conn(0x7f65b006e800 0x7f65b0070cb0 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7f65c8006c90 tx=0x7f65c80091d0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 10 monclient: shutdown 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:20.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 msgr2=0x7f65cc0a7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0a7e40 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f65c40099f0 tx=0x7f65c4037000 comp rx=0 tx=0).stop 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 shutdown_connections 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65cc13e600 0x7f65cc0a7e40 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0b1250 0x7f65cc0a7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 --2- 172.21.15.53:0/1185273438 >> v2:172.21.15.104:6816/32804 conn(0x7f65b006e800 0x7f65b0070cb0 unknown :-1 s=CLOSED pgs=3688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 --2- 172.21.15.53:0/1185273438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65cc0b0e80 0x7f65cc0a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 >> 172.21.15.53:0/1185273438 conn(0x7f65cc062d90 msgr2=0x7f65cc13d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 shutdown_connections 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 -- 172.21.15.53:0/1185273438 wait complete. 2024-09-15T02:58:20.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.277+0000 7f65d375d640 1 librados: shutdown 2024-09-15T02:58:20.288 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":1145,"stamp":"2024-09-15T02:58:19.540071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2471344,"kb_used_data":50876,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752503376,"statfs":{"total":773094113280,"available":770563457024,"internally_reserved":0,"allocated":52097024,"data_stored":25969454,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005021"},"pg_stats":[{"pgid":"12.10","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539924+0000","last_change":"2024-09-15T02:44:21.592595+0000","last_active":"2024-09-15T02:58:16.539924+0000","last_peered":"2024-09-15T02:58:16.539924+0000","last_clean":"2024-09-15T02:58:16.539924+0000","last_became_active":"2024-09-15T02:44:21.589224+0000","last_became_peered":"2024-09-15T02:44:21.589224+0000","last_unstale":"2024-09-15T02:58:16.539924+0000","last_undegraded":"2024-09-15T02:58:16.539924+0000","last_fullsized":"2024-09-15T02:58:16.539924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:30:34.422407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836595+0000","last_change":"2024-09-15T02:44:21.472953+0000","last_active":"2024-09-15T02:58:18.836595+0000","last_peered":"2024-09-15T02:58:18.836595+0000","last_clean":"2024-09-15T02:58:18.836595+0000","last_became_active":"2024-09-15T02:44:21.472608+0000","last_became_peered":"2024-09-15T02:44:21.472608+0000","last_unstale":"2024-09-15T02:58:18.836595+0000","last_undegraded":"2024-09-15T02:58:18.836595+0000","last_fullsized":"2024-09-15T02:58:18.836595+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:48.488509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.593941+0000","last_change":"2024-09-15T02:44:21.605637+0000","last_active":"2024-09-15T02:58:13.593941+0000","last_peered":"2024-09-15T02:58:13.593941+0000","last_clean":"2024-09-15T02:58:13.593941+0000","last_became_active":"2024-09-15T02:44:21.604946+0000","last_became_peered":"2024-09-15T02:44:21.604946+0000","last_unstale":"2024-09-15T02:58:13.593941+0000","last_undegraded":"2024-09-15T02:58:13.593941+0000","last_fullsized":"2024-09-15T02:58:13.593941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:21:30.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.538478+0000","last_change":"2024-09-15T02:44:21.631801+0000","last_active":"2024-09-15T02:58:15.538478+0000","last_peered":"2024-09-15T02:58:15.538478+0000","last_clean":"2024-09-15T02:58:15.538478+0000","last_became_active":"2024-09-15T02:44:21.621686+0000","last_became_peered":"2024-09-15T02:44:21.621686+0000","last_unstale":"2024-09-15T02:58:15.538478+0000","last_undegraded":"2024-09-15T02:58:15.538478+0000","last_fullsized":"2024-09-15T02:58:15.538478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:21:17.555939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836680+0000","last_change":"2024-09-15T02:44:21.559633+0000","last_active":"2024-09-15T02:58:18.836680+0000","last_peered":"2024-09-15T02:58:18.836680+0000","last_clean":"2024-09-15T02:58:18.836680+0000","last_became_active":"2024-09-15T02:44:21.558951+0000","last_became_peered":"2024-09-15T02:44:21.558951+0000","last_unstale":"2024-09-15T02:58:18.836680+0000","last_undegraded":"2024-09-15T02:58:18.836680+0000","last_fullsized":"2024-09-15T02:58:18.836680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:27.075112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236232+0000","last_change":"2024-09-15T02:44:21.593138+0000","last_active":"2024-09-15T02:58:17.236232+0000","last_peered":"2024-09-15T02:58:17.236232+0000","last_clean":"2024-09-15T02:58:17.236232+0000","last_became_active":"2024-09-15T02:44:21.592665+0000","last_became_peered":"2024-09-15T02:44:21.592665+0000","last_unstale":"2024-09-15T02:58:17.236232+0000","last_undegraded":"2024-09-15T02:58:17.236232+0000","last_fullsized":"2024-09-15T02:58:17.236232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:20.534135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235717+0000","last_change":"2024-09-15T02:44:21.537510+0000","last_active":"2024-09-15T02:58:17.235717+0000","last_peered":"2024-09-15T02:58:17.235717+0000","last_clean":"2024-09-15T02:58:17.235717+0000","last_became_active":"2024-09-15T02:44:21.537279+0000","last_became_peered":"2024-09-15T02:44:21.537279+0000","last_unstale":"2024-09-15T02:58:17.235717+0000","last_undegraded":"2024-09-15T02:58:17.235717+0000","last_fullsized":"2024-09-15T02:58:17.235717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:55:56.290343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235427+0000","last_change":"2024-09-15T02:44:21.613091+0000","last_active":"2024-09-15T02:58:17.235427+0000","last_peered":"2024-09-15T02:58:17.235427+0000","last_clean":"2024-09-15T02:58:17.235427+0000","last_became_active":"2024-09-15T02:44:21.612841+0000","last_became_peered":"2024-09-15T02:44:21.612841+0000","last_unstale":"2024-09-15T02:58:17.235427+0000","last_undegraded":"2024-09-15T02:58:17.235427+0000","last_fullsized":"2024-09-15T02:58:17.235427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:28:24.283127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537152+0000","last_change":"2024-09-15T02:44:21.543344+0000","last_active":"2024-09-15T02:58:15.537152+0000","last_peered":"2024-09-15T02:58:15.537152+0000","last_clean":"2024-09-15T02:58:15.537152+0000","last_became_active":"2024-09-15T02:44:21.522860+0000","last_became_peered":"2024-09-15T02:44:21.522860+0000","last_unstale":"2024-09-15T02:58:15.537152+0000","last_undegraded":"2024-09-15T02:58:15.537152+0000","last_fullsized":"2024-09-15T02:58:15.537152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:30.830504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541359+0000","last_change":"2024-09-15T02:44:21.538578+0000","last_active":"2024-09-15T02:58:16.541359+0000","last_peered":"2024-09-15T02:58:16.541359+0000","last_clean":"2024-09-15T02:58:16.541359+0000","last_became_active":"2024-09-15T02:44:21.535334+0000","last_became_peered":"2024-09-15T02:44:21.535334+0000","last_unstale":"2024-09-15T02:58:16.541359+0000","last_undegraded":"2024-09-15T02:58:16.541359+0000","last_fullsized":"2024-09-15T02:58:16.541359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:38:25.943125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542872+0000","last_change":"2024-09-15T02:44:21.614405+0000","last_active":"2024-09-15T02:58:17.542872+0000","last_peered":"2024-09-15T02:58:17.542872+0000","last_clean":"2024-09-15T02:58:17.542872+0000","last_became_active":"2024-09-15T02:44:21.614186+0000","last_became_peered":"2024-09-15T02:44:21.614186+0000","last_unstale":"2024-09-15T02:58:17.542872+0000","last_undegraded":"2024-09-15T02:58:17.542872+0000","last_fullsized":"2024-09-15T02:58:17.542872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:52.698525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542738+0000","last_change":"2024-09-15T02:44:21.527859+0000","last_active":"2024-09-15T02:58:17.542738+0000","last_peered":"2024-09-15T02:58:17.542738+0000","last_clean":"2024-09-15T02:58:17.542738+0000","last_became_active":"2024-09-15T02:44:21.527331+0000","last_became_peered":"2024-09-15T02:44:21.527331+0000","last_unstale":"2024-09-15T02:58:17.542738+0000","last_undegraded":"2024-09-15T02:58:17.542738+0000","last_fullsized":"2024-09-15T02:58:17.542738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:55:15.776872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":653,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235679+0000","last_change":"2024-09-15T02:43:23.626724+0000","last_active":"2024-09-15T02:58:17.235679+0000","last_peered":"2024-09-15T02:58:17.235679+0000","last_clean":"2024-09-15T02:58:17.235679+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:58:17.235679+0000","last_undegraded":"2024-09-15T02:58:17.235679+0000","last_fullsized":"2024-09-15T02:58:17.235679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:48.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540373+0000","last_change":"2024-09-15T02:44:21.539798+0000","last_active":"2024-09-15T02:58:17.540373+0000","last_peered":"2024-09-15T02:58:17.540373+0000","last_clean":"2024-09-15T02:58:17.540373+0000","last_became_active":"2024-09-15T02:44:21.530274+0000","last_became_peered":"2024-09-15T02:44:21.530274+0000","last_unstale":"2024-09-15T02:58:17.540373+0000","last_undegraded":"2024-09-15T02:58:17.540373+0000","last_fullsized":"2024-09-15T02:58:17.540373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:24:08.051420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":655,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537819+0000","last_change":"2024-09-15T02:43:23.426048+0000","last_active":"2024-09-15T02:58:15.537819+0000","last_peered":"2024-09-15T02:58:15.537819+0000","last_clean":"2024-09-15T02:58:15.537819+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:58:15.537819+0000","last_undegraded":"2024-09-15T02:58:15.537819+0000","last_fullsized":"2024-09-15T02:58:15.537819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:19.460873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.594062+0000","last_change":"2024-09-15T02:44:21.579868+0000","last_active":"2024-09-15T02:58:13.594062+0000","last_peered":"2024-09-15T02:58:13.594062+0000","last_clean":"2024-09-15T02:58:13.594062+0000","last_became_active":"2024-09-15T02:44:21.579154+0000","last_became_peered":"2024-09-15T02:44:21.579154+0000","last_unstale":"2024-09-15T02:58:13.594062+0000","last_undegraded":"2024-09-15T02:58:13.594062+0000","last_fullsized":"2024-09-15T02:58:13.594062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:45:09.149484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":650,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914260+0000","last_change":"2024-09-15T02:43:23.420483+0000","last_active":"2024-09-15T02:58:16.914260+0000","last_peered":"2024-09-15T02:58:16.914260+0000","last_clean":"2024-09-15T02:58:16.914260+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:58:16.914260+0000","last_undegraded":"2024-09-15T02:58:16.914260+0000","last_fullsized":"2024-09-15T02:58:16.914260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:03:14.233618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":653,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T02:58:13.594285+0000","last_change":"2024-09-15T02:44:21.537181+0000","last_active":"2024-09-15T02:58:13.594285+0000","last_peered":"2024-09-15T02:58:13.594285+0000","last_clean":"2024-09-15T02:58:13.594285+0000","last_became_active":"2024-09-15T02:44:21.533501+0000","last_became_peered":"2024-09-15T02:44:21.533501+0000","last_unstale":"2024-09-15T02:58:13.594285+0000","last_undegraded":"2024-09-15T02:58:13.594285+0000","last_fullsized":"2024-09-15T02:58:13.594285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:42.258617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":655,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537074+0000","last_change":"2024-09-15T02:43:23.426071+0000","last_active":"2024-09-15T02:58:15.537074+0000","last_peered":"2024-09-15T02:58:15.537074+0000","last_clean":"2024-09-15T02:58:15.537074+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:58:15.537074+0000","last_undegraded":"2024-09-15T02:58:15.537074+0000","last_fullsized":"2024-09-15T02:58:15.537074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:00:03.954753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540360+0000","last_change":"2024-09-15T02:44:21.561913+0000","last_active":"2024-09-15T02:58:17.540360+0000","last_peered":"2024-09-15T02:58:17.540360+0000","last_clean":"2024-09-15T02:58:17.540360+0000","last_became_active":"2024-09-15T02:44:21.560425+0000","last_became_peered":"2024-09-15T02:44:21.560425+0000","last_unstale":"2024-09-15T02:58:17.540360+0000","last_undegraded":"2024-09-15T02:58:17.540360+0000","last_fullsized":"2024-09-15T02:58:17.540360+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:40:57.301643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":658,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836590+0000","last_change":"2024-09-15T02:43:23.423983+0000","last_active":"2024-09-15T02:58:18.836590+0000","last_peered":"2024-09-15T02:58:18.836590+0000","last_clean":"2024-09-15T02:58:18.836590+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:58:18.836590+0000","last_undegraded":"2024-09-15T02:58:18.836590+0000","last_fullsized":"2024-09-15T02:58:18.836590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:45.683030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539798+0000","last_change":"2024-09-15T02:44:21.500932+0000","last_active":"2024-09-15T02:58:16.539798+0000","last_peered":"2024-09-15T02:58:16.539798+0000","last_clean":"2024-09-15T02:58:16.539798+0000","last_became_active":"2024-09-15T02:44:21.500646+0000","last_became_peered":"2024-09-15T02:44:21.500646+0000","last_unstale":"2024-09-15T02:58:16.539798+0000","last_undegraded":"2024-09-15T02:58:16.539798+0000","last_fullsized":"2024-09-15T02:58:16.539798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:08.733289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":663,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540358+0000","last_change":"2024-09-15T02:43:23.419938+0000","last_active":"2024-09-15T02:58:17.540358+0000","last_peered":"2024-09-15T02:58:17.540358+0000","last_clean":"2024-09-15T02:58:17.540358+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:58:17.540358+0000","last_undegraded":"2024-09-15T02:58:17.540358+0000","last_fullsized":"2024-09-15T02:58:17.540358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:09:35.753407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235268+0000","last_change":"2024-09-15T02:44:21.504909+0000","last_active":"2024-09-15T02:58:17.235268+0000","last_peered":"2024-09-15T02:58:17.235268+0000","last_clean":"2024-09-15T02:58:17.235268+0000","last_became_active":"2024-09-15T02:44:21.504665+0000","last_became_peered":"2024-09-15T02:44:21.504665+0000","last_unstale":"2024-09-15T02:58:17.235268+0000","last_undegraded":"2024-09-15T02:58:17.235268+0000","last_fullsized":"2024-09-15T02:58:17.235268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:17:58.326032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":649,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913924+0000","last_change":"2024-09-15T02:43:23.420430+0000","last_active":"2024-09-15T02:58:16.913924+0000","last_peered":"2024-09-15T02:58:16.913924+0000","last_clean":"2024-09-15T02:58:16.913924+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:58:16.913924+0000","last_undegraded":"2024-09-15T02:58:16.913924+0000","last_fullsized":"2024-09-15T02:58:16.913924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:05:17.337850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.543353+0000","last_change":"2024-09-15T02:44:21.645181+0000","last_active":"2024-09-15T02:58:16.543353+0000","last_peered":"2024-09-15T02:58:16.543353+0000","last_clean":"2024-09-15T02:58:16.543353+0000","last_became_active":"2024-09-15T02:44:21.644939+0000","last_became_peered":"2024-09-15T02:44:21.644939+0000","last_unstale":"2024-09-15T02:58:16.543353+0000","last_undegraded":"2024-09-15T02:58:16.543353+0000","last_fullsized":"2024-09-15T02:58:16.543353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:22:20.773021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":657,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836538+0000","last_change":"2024-09-15T02:43:23.423477+0000","last_active":"2024-09-15T02:58:18.836538+0000","last_peered":"2024-09-15T02:58:18.836538+0000","last_clean":"2024-09-15T02:58:18.836538+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:58:18.836538+0000","last_undegraded":"2024-09-15T02:58:18.836538+0000","last_fullsized":"2024-09-15T02:58:18.836538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:05:30.605263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":735,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913790+0000","last_change":"2024-09-15T02:43:20.387731+0000","last_active":"2024-09-15T02:58:16.913790+0000","last_peered":"2024-09-15T02:58:16.913790+0000","last_clean":"2024-09-15T02:58:16.913790+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:58:16.913790+0000","last_undegraded":"2024-09-15T02:58:16.913790+0000","last_fullsized":"2024-09-15T02:58:16.913790+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_clean_scrub_stamp":"2024-09-15T02:43:19.374225+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:33:29.816927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":639,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914166+0000","last_change":"2024-09-15T02:44:21.629491+0000","last_active":"2024-09-15T02:58:16.914166+0000","last_peered":"2024-09-15T02:58:16.914166+0000","last_clean":"2024-09-15T02:58:16.914166+0000","last_became_active":"2024-09-15T02:44:21.629239+0000","last_became_peered":"2024-09-15T02:44:21.629239+0000","last_unstale":"2024-09-15T02:58:16.914166+0000","last_undegraded":"2024-09-15T02:58:16.914166+0000","last_fullsized":"2024-09-15T02:58:16.914166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:56.354264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.538060+0000","last_change":"2024-09-15T02:44:21.573593+0000","last_active":"2024-09-15T02:58:15.538060+0000","last_peered":"2024-09-15T02:58:15.538060+0000","last_clean":"2024-09-15T02:58:15.538060+0000","last_became_active":"2024-09-15T02:44:21.571415+0000","last_became_peered":"2024-09-15T02:44:21.571415+0000","last_unstale":"2024-09-15T02:58:15.538060+0000","last_undegraded":"2024-09-15T02:58:15.538060+0000","last_fullsized":"2024-09-15T02:58:15.538060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:12.880270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-15T02:58:15.537934+0000","last_change":"2024-09-15T02:44:21.629604+0000","last_active":"2024-09-15T02:58:15.537934+0000","last_peered":"2024-09-15T02:58:15.537934+0000","last_clean":"2024-09-15T02:58:15.537934+0000","last_became_active":"2024-09-15T02:44:21.628817+0000","last_became_peered":"2024-09-15T02:44:21.628817+0000","last_unstale":"2024-09-15T02:58:15.537934+0000","last_undegraded":"2024-09-15T02:58:15.537934+0000","last_fullsized":"2024-09-15T02:58:15.537934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:52:41.637023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236078+0000","last_change":"2024-09-15T02:44:21.583579+0000","last_active":"2024-09-15T02:58:17.236078+0000","last_peered":"2024-09-15T02:58:17.236078+0000","last_clean":"2024-09-15T02:58:17.236078+0000","last_became_active":"2024-09-15T02:44:21.583340+0000","last_became_peered":"2024-09-15T02:44:21.583340+0000","last_unstale":"2024-09-15T02:58:17.236078+0000","last_undegraded":"2024-09-15T02:58:17.236078+0000","last_fullsized":"2024-09-15T02:58:17.236078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:01:58.119370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235646+0000","last_change":"2024-09-15T02:44:21.616190+0000","last_active":"2024-09-15T02:58:17.235646+0000","last_peered":"2024-09-15T02:58:17.235646+0000","last_clean":"2024-09-15T02:58:17.235646+0000","last_became_active":"2024-09-15T02:44:21.615991+0000","last_became_peered":"2024-09-15T02:44:21.615991+0000","last_unstale":"2024-09-15T02:58:17.235646+0000","last_undegraded":"2024-09-15T02:58:17.235646+0000","last_fullsized":"2024-09-15T02:58:17.235646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:25:27.900125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539926+0000","last_change":"2024-09-15T02:44:21.608529+0000","last_active":"2024-09-15T02:58:16.539926+0000","last_peered":"2024-09-15T02:58:16.539926+0000","last_clean":"2024-09-15T02:58:16.539926+0000","last_became_active":"2024-09-15T02:44:21.588781+0000","last_became_peered":"2024-09-15T02:44:21.588781+0000","last_unstale":"2024-09-15T02:58:16.539926+0000","last_undegraded":"2024-09-15T02:58:16.539926+0000","last_fullsized":"2024-09-15T02:58:16.539926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:49.754802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.614367+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.613899+0000","last_became_peered":"2024-09-15T02:44:21.613899+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:18:47.868539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1","version":"22'7","reported_seq":655,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235332+0000","last_change":"2024-09-15T02:44:21.595390+0000","last_active":"2024-09-15T02:58:17.235332+0000","last_peered":"2024-09-15T02:58:17.235332+0000","last_clean":"2024-09-15T02:58:17.235332+0000","last_became_active":"2024-09-15T02:44:21.594670+0000","last_became_peered":"2024-09-15T02:44:21.594670+0000","last_unstale":"2024-09-15T02:58:17.235332+0000","last_undegraded":"2024-09-15T02:58:17.235332+0000","last_fullsized":"2024-09-15T02:58:17.235332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:52:07.118143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.536146+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.535920+0000","last_became_peered":"2024-09-15T02:44:21.535920+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.095882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235267+0000","last_change":"2024-09-15T02:44:21.578890+0000","last_active":"2024-09-15T02:58:17.235267+0000","last_peered":"2024-09-15T02:58:17.235267+0000","last_clean":"2024-09-15T02:58:17.235267+0000","last_became_active":"2024-09-15T02:44:21.578571+0000","last_became_peered":"2024-09-15T02:44:21.578571+0000","last_unstale":"2024-09-15T02:58:17.235267+0000","last_undegraded":"2024-09-15T02:58:17.235267+0000","last_fullsized":"2024-09-15T02:58:17.235267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:01.395702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542928+0000","last_change":"2024-09-15T02:44:21.556333+0000","last_active":"2024-09-15T02:58:17.542928+0000","last_peered":"2024-09-15T02:58:17.542928+0000","last_clean":"2024-09-15T02:58:17.542928+0000","last_became_active":"2024-09-15T02:44:21.556078+0000","last_became_peered":"2024-09-15T02:44:21.556078+0000","last_unstale":"2024-09-15T02:58:17.542928+0000","last_undegraded":"2024-09-15T02:58:17.542928+0000","last_fullsized":"2024-09-15T02:58:17.542928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:35:40.520618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541145+0000","last_change":"2024-09-15T02:44:21.583194+0000","last_active":"2024-09-15T02:58:16.541145+0000","last_peered":"2024-09-15T02:58:16.541145+0000","last_clean":"2024-09-15T02:58:16.541145+0000","last_became_active":"2024-09-15T02:44:21.582718+0000","last_became_peered":"2024-09-15T02:44:21.582718+0000","last_unstale":"2024-09-15T02:58:16.541145+0000","last_undegraded":"2024-09-15T02:58:16.541145+0000","last_fullsized":"2024-09-15T02:58:16.541145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:48:37.528157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.540867+0000","last_change":"2024-09-15T02:44:21.584252+0000","last_active":"2024-09-15T02:58:16.540867+0000","last_peered":"2024-09-15T02:58:16.540867+0000","last_clean":"2024-09-15T02:58:16.540867+0000","last_became_active":"2024-09-15T02:44:21.584028+0000","last_became_peered":"2024-09-15T02:44:21.584028+0000","last_unstale":"2024-09-15T02:58:16.540867+0000","last_undegraded":"2024-09-15T02:58:16.540867+0000","last_fullsized":"2024-09-15T02:58:16.540867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:14.221941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575031,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312068,"kb_used_data":6388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059772,"statfs":{"total":96636764160,"available":96317206528,"internally_reserved":0,"allocated":6541312,"data_stored":3635868,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.69399999999999995,"5min":0.75,"15min":1.3959999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.28100000000000003},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.80500000000000005},"last":0.76500000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.55300000000000005},"min":{"1min":0.46100000000000002,"5min":0.33600000000000002,"15min":0.23400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.94599999999999995},"last":0.745},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59799999999999998,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.27300000000000002},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":1.4430000000000001},"last":0.67900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60799999999999998,"15min":0.56200000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.27000000000000002},"max":{"1min":0.745,"5min":0.78600000000000003,"15min":0.95899999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.20300000000000001},"max":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":1.478},"last":0.80300000000000005}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57599999999999996,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.28299999999999997},"max":{"1min":0.72699999999999998,"5min":0.83699999999999997,"15min":2.4300000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.57299999999999995},"min":{"1min":0.502,"5min":0.40600000000000003,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.89700000000000002,"15min":1.6120000000000001},"last":0.71599999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45100000000000001,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.24199999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.755},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.28000000000000003},"max":{"1min":0.70599999999999996,"5min":0.83399999999999996,"15min":0.95899999999999996},"last":0.626}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":0.45400000000000001},"min":{"1min":0.35499999999999998,"5min":0.30099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72899999999999998,"15min":0.88900000000000001},"last":0.502},{"interface":"front","average":{"1min":0.44,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":2.2400000000000002},"last":0.57699999999999996}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.77600000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.52500000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.79300000000000004,"5min":0.89700000000000002,"15min":0.92800000000000005},"last":0.55000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312012,"kb_used_data":6340,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059828,"statfs":{"total":96636764160,"available":96317263872,"internally_reserved":0,"allocated":6492160,"data_stored":3106945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.34599999999999997,"15min":0.27000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":5.0250000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.874},"min":{"1min":0.46600000000000003,"5min":0.37,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":0.85699999999999998,"15min":71.516999999999996},"last":0.45400000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.94699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.25900000000000001},"max":{"1min":0.69399999999999995,"5min":0.84199999999999997,"15min":93.381},"last":0.5},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56100000000000005,"15min":0.86399999999999999},"min":{"1min":0.39300000000000002,"5min":0.33500000000000002,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.90900000000000003,"15min":79.905000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.66400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.93899999999999995,"15min":29.305},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.318},"max":{"1min":0.66500000000000004,"5min":0.85499999999999998,"15min":9.8510000000000009},"last":0.54800000000000004}]},{"osd":3,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.69499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.255},"max":{"1min":0.75900000000000001,"5min":0.96599999999999997,"15min":34.877000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.29299999999999998},"max":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":31.657},"last":0.51000000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.51000000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.20300000000000001},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":20.212},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.442,"15min":0.46300000000000002},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.64700000000000002,"5min":0.71599999999999997,"15min":5.6909999999999998},"last":0.35999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.215},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80000000000000004},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.248},"max":{"1min":0.71499999999999997,"5min":0.81799999999999995,"15min":3.012},"last":0.58699999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47099999999999997,"15min":0.59499999999999997},"min":{"1min":0.35099999999999998,"5min":0.32500000000000001,"15min":0.24299999999999999},"max":{"1min":0.68999999999999995,"5min":0.71299999999999997,"15min":28.756},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.48899999999999999,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.32000000000000001,"15min":0.249},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":27.602},"last":0.436}]}]},{"osd":5,"up_from":13,"seq":55834575032,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":6352,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":6504448,"data_stored":3112540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.69899999999999995},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":17.704999999999998},"last":0.435},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.621,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.78100000000000003,"5min":0.872,"15min":17.465},"last":0.54900000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.22900000000000001},"max":{"1min":0.81999999999999995,"5min":0.83999999999999997,"15min":7.1710000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.42999999999999999,"5min":0.33300000000000002,"15min":0.27700000000000002},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":6.7999999999999998},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1000000000000001,"15min":8.2789999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.60299999999999998},"min":{"1min":0.51600000000000001,"5min":0.33800000000000002,"15min":0.29099999999999998},"max":{"1min":0.79300000000000004,"5min":1.133,"15min":3.0059999999999998},"last":0.61299999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.628},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.25700000000000001},"max":{"1min":0.81299999999999994,"5min":0.872,"15min":7.8090000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63600000000000001,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.38200000000000001,"15min":0.28599999999999998},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":8.6829999999999998},"last":0.72399999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46300000000000002,"15min":0.44600000000000001},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.216},"max":{"1min":0.68100000000000005,"5min":0.82099999999999995,"15min":0.98699999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.23000000000000001},"max":{"1min":0.68300000000000005,"5min":0.78000000000000003,"15min":2.0750000000000002},"last":0.63200000000000001}]},{"osd":6,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.22800000000000001},"max":{"1min":0.67000000000000004,"5min":0.81799999999999995,"15min":3.4420000000000002},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.39100000000000001,"5min":0.33300000000000002,"15min":0.25},"max":{"1min":0.68899999999999995,"5min":0.86299999999999999,"15min":3.0510000000000002},"last":0.65200000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.253},"max":{"1min":0.69699999999999995,"5min":0.79400000000000004,"15min":2.9870000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.94399999999999995,"15min":1.337},"last":0.78200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307884,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063956,"statfs":{"total":96636764160,"available":96321490944,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.86299999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.316},"max":{"1min":0.72099999999999997,"5min":0.873,"15min":0.93100000000000005},"last":0.72699999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.69699999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.32800000000000001},"max":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.379,"15min":0.223},"max":{"1min":0.73699999999999999,"5min":0.89400000000000002,"15min":0.94199999999999995},"last":0.754},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.83499999999999996}]},{"osd":3,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.47799999999999998,"5min":0.36699999999999999,"15min":0.26800000000000002},"max":{"1min":0.82499999999999996,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60399999999999998,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.36099999999999999},"max":{"1min":0.752,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.23200000000000001},"max":{"1min":0.67400000000000004,"5min":0.69699999999999995,"15min":0.81599999999999995},"last":0.47999999999999998}]},{"osd":6,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.497},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42099999999999999,"15min":0.44700000000000001},"min":{"1min":0.316,"5min":0.31,"15min":0.251},"max":{"1min":0.505,"5min":0.65700000000000003,"15min":0.69699999999999995},"last":0.59599999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.78600000000000003,"5min":1.48,"15min":1.48},"last":0.439},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.49199999999999999,"15min":0.50600000000000001},"min":{"1min":0.373,"5min":0.31900000000000001,"15min":0.28000000000000003},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.81799999999999995},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575031,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307852,"kb_used_data":6336,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063988,"statfs":{"total":96636764160,"available":96321523712,"internally_reserved":0,"allocated":6488064,"data_stored":3127885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.371,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.52200000000000002,"5min":0.72399999999999998,"15min":29.562999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47899999999999998,"15min":0.44400000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":0.88600000000000001,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":0.307}]},{"osd":1,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.57199999999999995},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.19600000000000001},"max":{"1min":0.64400000000000002,"5min":1.6859999999999999,"15min":35.128},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.439,"15min":0.56999999999999995},"min":{"1min":0.28199999999999997,"5min":0.26600000000000001,"15min":0.24299999999999999},"max":{"1min":0.54600000000000004,"5min":1.252,"15min":29.631},"last":0.33400000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46200000000000002,"15min":0.60399999999999998},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.23000000000000001},"max":{"1min":0.622,"5min":0.90000000000000002,"15min":35.159999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.61099999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.63900000000000001,"5min":0.88600000000000001,"15min":29.603000000000002},"last":0.432}]},{"osd":4,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63,"15min":0.65900000000000003},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":0.80200000000000005,"5min":0.93899999999999995,"15min":7.3499999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64100000000000001,"15min":0.79300000000000004},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":1.0569999999999999,"15min":35.188000000000002},"last":0.60399999999999998}]},{"osd":5,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.72699999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84999999999999998,"5min":1.165,"15min":23.082999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":0.93899999999999995,"5min":1.323,"15min":7.4020000000000001},"last":0.58599999999999997}]},{"osd":6,"last update":"Sun Sep 15 02:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67300000000000004,"15min":0.78900000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.373},"max":{"1min":0.97699999999999998,"5min":1.365,"15min":35.061999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64600000000000002,"15min":0.74399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44},"max":{"1min":0.90900000000000003,"5min":1.018,"15min":23.129000000000001},"last":0.56799999999999995}]},{"osd":7,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.5,"5min":0.49199999999999999,"15min":0.36699999999999999},"max":{"1min":0.88,"5min":1.1060000000000001,"15min":2.1840000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.80900000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.42099999999999999},"max":{"1min":0.81899999999999995,"5min":0.999,"15min":35.100999999999999},"last":0.55300000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575031,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312044,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059796,"statfs":{"total":96636764160,"available":96317231104,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.253,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":0.55200000000000005,"5min":0.65000000000000002,"15min":1.2370000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":0.57199999999999995,"5min":0.747,"15min":1.1799999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.27100000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59099999999999997,"5min":0.70999999999999996,"15min":1.319},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.73799999999999999,"15min":1.347},"last":0.52700000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.34899999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.55900000000000005,"5min":0.66700000000000004,"15min":1.411},"last":0.502},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.34599999999999997,"5min":0.29199999999999998,"15min":0.252},"max":{"1min":0.622,"5min":0.67500000000000004,"15min":2.9340000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.73099999999999998,"5min":0.95499999999999996,"15min":1.298},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.61699999999999999},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.875,"15min":1.1319999999999999},"last":0.625}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.315},"max":{"1min":0.81999999999999995,"5min":1,"15min":2.9079999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.49199999999999999,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.94999999999999996,"15min":1.367},"last":0.79900000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.008,"15min":1.2050000000000001},"last":0.753},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.96999999999999997,"15min":2.9649999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.50800000000000001,"5min":0.43099999999999999,"15min":0.31900000000000001},"max":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":1.3240000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.44600000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":0.82099999999999995,"5min":0.95199999999999996,"15min":1.167},"last":0.81799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303692,"kb_used_data":6340,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068148,"statfs":{"total":96636764160,"available":96325783552,"internally_reserved":0,"allocated":6492160,"data_stored":3106959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.38900000000000001,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.17899999999999999},"max":{"1min":0.70399999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51800000000000002,"5min":0.65100000000000002,"15min":0.93100000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56000000000000005,"5min":1.405,"15min":1.405},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.441},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.219},"max":{"1min":0.64700000000000002,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.249},"max":{"1min":0.81000000000000005,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.67100000000000004,"5min":2.8490000000000002,"15min":2.8490000000000002},"last":0.50700000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63200000000000001,"15min":0.59999999999999998},"min":{"1min":0.48899999999999999,"5min":0.379,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":1.581,"15min":1.581},"last":0.875},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63800000000000001,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.34899999999999998,"15min":0.29299999999999998},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.95499999999999996},"last":0.65900000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.64200000000000002},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.33300000000000002},"max":{"1min":0.90200000000000002,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.65800000000000003},"min":{"1min":0.56200000000000006,"5min":0.433,"15min":0.371},"max":{"1min":1.004,"5min":1.532,"15min":1.532},"last":0.77100000000000002}]},{"osd":6,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.40400000000000003,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.70299999999999996,"15min":0.66300000000000003},"min":{"1min":0.60899999999999999,"5min":0.46800000000000003,"15min":0.33000000000000002},"max":{"1min":1.0509999999999999,"5min":1.498,"15min":1.498},"last":0.82899999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.53500000000000003,"5min":0.35999999999999999,"15min":0.35699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.32500000000000001},"max":{"1min":0.97399999999999998,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.72999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307920,"kb_used_data":6400,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063920,"statfs":{"total":96636764160,"available":96321454080,"internally_reserved":0,"allocated":6553600,"data_stored":3641449,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.503},"min":{"1min":0.251,"5min":0.24199999999999999,"15min":0.182},"max":{"1min":0.64800000000000002,"5min":0.68500000000000005,"15min":14.630000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46999999999999997,"15min":0.45100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.222},"max":{"1min":0.69299999999999995,"5min":0.85399999999999998,"15min":2.3570000000000002},"last":0.60699999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44500000000000001,"15min":0.47499999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.20200000000000001},"max":{"1min":0.67000000000000004,"5min":0.93899999999999995,"15min":14.679},"last":0.503},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":0.68600000000000005,"15min":1.806},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.21099999999999999},"max":{"1min":0.504,"5min":0.73799999999999999,"15min":2.835},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.48899999999999999,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.26300000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":14.558999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64900000000000002,"15min":0.60499999999999998},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.253},"max":{"1min":0.85599999999999998,"5min":0.98599999999999999,"15min":1.5549999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65300000000000002,"15min":0.61699999999999999},"min":{"1min":0.38900000000000001,"5min":0.33200000000000002,"15min":0.30199999999999999},"max":{"1min":0.79000000000000004,"5min":0.94699999999999995,"15min":1.3120000000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63400000000000001,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.32800000000000001},"max":{"1min":0.878,"5min":0.998,"15min":1.1579999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.34300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":7.2969999999999997},"last":0.67500000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68200000000000005,"15min":0.70599999999999996},"min":{"1min":0.55700000000000005,"5min":0.48599999999999999,"15min":0.35199999999999998},"max":{"1min":0.97499999999999998,"5min":1.0129999999999999,"15min":14.6},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69599999999999995,"15min":0.65400000000000003},"min":{"1min":0.53200000000000003,"5min":0.40999999999999998,"15min":0.35399999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.79600000000000004}]},{"osd":7,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67900000000000005,"15min":0.64000000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.36699999999999999},"max":{"1min":0.95399999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.34300000000000003},"max":{"1min":0.97899999999999998,"5min":1.0029999999999999,"15min":1.093},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:58:20.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:58:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.387+0000 7f3656b9d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:20.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.387+0000 7f3656b9d640 10 monclient: build_initial_monmap 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.387+0000 7f3656b9d640 10 monclient: monmap: 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:20.388659+0000 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:20.388659+0000 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:20.389 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.388+0000 7f3656b9d640 1 Processor -- start 2024-09-15T02:58:20.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.388+0000 7f3656b9d640 1 -- start start 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.388+0000 7f3656b9d640 10 monclient: init 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3650140f70 0x7f36500b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3650140f70 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3650141340 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500b0a80 0x7f36500b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f36500b0a80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365013ddd0 con 0x7f3650141340 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365013e970 con 0x7f3650140f70 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36500a6c00 con 0x7f36500b0a80 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3656b9d640 10 monclient(hunting): authenticate will time out at 1636.959457s 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500b0a80 0x7f36500b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3650140f70 0x7f36500b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500b0a80 0x7f36500b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56214/0 (socket says 172.21.15.53:56214) 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 1 -- 172.21.15.53:0/828678049 learned_addr learned my addr 172.21.15.53:0/828678049 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): get_auth_request con 0x7f3650140f70 auth_method 0 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f364ffff640 1 --1- 172.21.15.53:0/828678049 >> v1:172.21.15.53:6789/0 conn(0x7f3650141340 0x7f365013e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59206/0 (socket says 172.21.15.53:59206) 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 10 monclient(hunting): get_auth_request con 0x7f36500b0a80 auth_method 0 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient(hunting): handle_auth_done global_id 26235 payload 274 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 1 -- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500b0a80 msgr2=0x7f36500b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 1 --2- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500b0a80 0x7f36500b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 1 -- 172.21.15.53:0/828678049 >> v1:172.21.15.53:6789/0 conn(0x7f3650141340 legacy=0x7f365013e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient: _finish_hunting 0 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 1 monclient: found mon.noname-a 2024-09-15T02:58:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f364f7fe640 1 -- 172.21.15.53:0/828678049 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311250268 0 0) 0x7f365013ddd0 con 0x7f3650141340 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f3654939640 1 -- 172.21.15.53:0/828678049 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500a5b30 con 0x7f3650140f70 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.389+0000 7f365513a640 1 --2- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500b0a80 0x7f36500b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f3654939640 1 --2- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650140f70 0x7f36500b00f0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 crypto rx=0x7f3640009880 tx=0x7f36400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46d22b0e04f0e5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 1 -- 172.21.15.53:0/828678049 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364002ace0 con 0x7f3650140f70 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: dump: 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 1 -- 172.21.15.53:0/828678049 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364002ae40 con 0x7f3650140f70 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f3656b9d640 5 monclient: authenticate success, global_id 26235 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f3656b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:20.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f3656b9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 1 -- 172.21.15.53:0/828678049 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640032670 con 0x7f3650140f70 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f3656b9d640 4 set_mon_vals no callback set 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.390+0000 7f364f7fe640 10 monclient: dump: 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.403 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 10 monclient: shutdown 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 -- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650140f70 msgr2=0x7f36500b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 --2- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650140f70 0x7f36500b00f0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 crypto rx=0x7f3640009880 tx=0x7f36400090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 -- 172.21.15.53:0/828678049 shutdown_connections 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 --2- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500b0a80 0x7f36500b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 --2- 172.21.15.53:0/828678049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650140f70 0x7f36500b00f0 unknown :-1 s=CLOSED pgs=5474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 -- 172.21.15.53:0/828678049 >> 172.21.15.53:0/828678049 conn(0x7f3650062d90 msgr2=0x7f3650063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 -- 172.21.15.53:0/828678049 shutdown_connections 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.391+0000 7f3656b9d640 1 -- 172.21.15.53:0/828678049 wait complete. 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 10 monclient: build_initial_monmap 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 10 monclient: monmap: 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 1 librados: starting msgr at 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 1 librados: starting objecter 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 1 Processor -- start 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 1 -- start start 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 1 librados: setting wanted keys 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 1 librados: calling monclient init 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.392+0000 7f3656b9d640 10 monclient: init 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36500b0a80 0x7f36500aab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f36500b0a80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3650140f70 0x7f36500a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3650140f70 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650141340 0x7f365013a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3650141340 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500a7550 con 0x7f3650141340 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500abeb0 con 0x7f36500b0a80 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36500aa230 con 0x7f3650140f70 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3656b9d640 10 monclient(hunting): authenticate will time out at 1636.963312s 2024-09-15T02:58:20.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650141340 0x7f365013a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3650140f70 0x7f36500a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650141340 0x7f365013a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52360/0 (socket says 172.21.15.53:52360) 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3650140f70 0x7f36500a9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56218/0 (socket says 172.21.15.53:56218) 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 1 -- 172.21.15.53:0/1930454631 learned_addr learned my addr 172.21.15.53:0/1930454631 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 10 monclient(hunting): get_auth_request con 0x7f3650141340 auth_method 0 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): get_auth_request con 0x7f3650140f70 auth_method 0 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36500b0a80 0x7f36500aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 10 monclient(hunting): get_auth_request con 0x7f36500b0a80 auth_method 0 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:20.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f3654939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient(hunting): handle_auth_done global_id 20333 payload 824 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 -- 172.21.15.53:0/1930454631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36500b0a80 msgr2=0x7f36500aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36500b0a80 0x7f36500aab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 -- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650141340 msgr2=0x7f365013a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650141340 0x7f365013a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 monclient: found mon.noname-c 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 -- 172.21.15.53:0/1930454631 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648009ae0 con 0x7f3650140f70 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f365513a640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650141340 0x7f365013a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364ffff640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3650140f70 0x7f36500a9cf0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f363800c460 tx=0x7f3638007e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364cff9640 10 client.?.objecter ms_handle_connect 0x7f3650140f70 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.393+0000 7f364cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 1 -- 172.21.15.53:0/1930454631 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36380072e0 con 0x7f3650140f70 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: dump: 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: _finish_auth 0 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 5 monclient: authenticate success, global_id 20333 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 1 -- 172.21.15.53:0/1930454631 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3638007440 con 0x7f3650140f70 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 1 -- 172.21.15.53:0/1930454631 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36380049f0 con 0x7f3650140f70 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 10 monclient: _renew_subs 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3640009390 con 0x7f3650140f70 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364e7fc640 4 set_mon_vals no callback set 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f364cff9640 10 monclient: dump: 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 10 client.20333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 10 monclient: _renew_subs 2024-09-15T02:58:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365013ac80 con 0x7f3650140f70 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.394+0000 7f3656b9d640 1 librados: init done 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.395+0000 7f3656b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.395+0000 7f3656b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.395+0000 7f3656b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.395+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36500a7320 con 0x7f3650140f70 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.395+0000 7f364cff9640 1 -- 172.21.15.53:0/1930454631 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f3638013400 con 0x7f3650140f70 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f364cff9640 1 --2- 172.21.15.53:0/1930454631 >> v2:172.21.15.104:6816/32804 conn(0x7f363006e800 0x7f3630070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f364cff9640 1 -- 172.21.15.53:0/1930454631 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f363808ad00 con 0x7f3650140f70 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f364cff9640 10 client.20333.objecter ms_dispatch 0x7f3650000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f364cff9640 3 client.20333.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f364cff9640 3 client.20333.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f364cff9640 20 client.20333.objecter dump_active .. 0 homeless 2024-09-15T02:58:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f3654939640 1 --2- 172.21.15.53:0/1930454631 >> v2:172.21.15.104:6816/32804 conn(0x7f363006e800 0x7f3630070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f3654939640 10 monclient: get_auth_request con 0x7f363006e800 auth_method 0 2024-09-15T02:58:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f3654939640 1 --2- 172.21.15.53:0/1930454631 >> v2:172.21.15.104:6816/32804 conn(0x7f363006e800 0x7f3630070cb0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7f3640034000 tx=0x7f3640032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.396+0000 7f364cff9640 10 client.20333.objecter ms_handle_connect 0x7f363006e800 2024-09-15T02:58:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.400+0000 7f364cff9640 1 -- 172.21.15.53:0/1930454631 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f36500a7320 con 0x7f3650140f70 2024-09-15T02:58:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.400+0000 7f364cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.400+0000 7f364cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:20.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.566+0000 7f3656b9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:58:20.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.566+0000 7f3656b9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:58:20.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.566+0000 7f3656b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:20.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.566+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36500a7320 con 0x7f3650140f70 2024-09-15T02:58:20.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.567+0000 7f364cff9640 1 -- 172.21.15.53:0/1930454631 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v702) v1 ==== 74+0+14637 (secure 0 0 0) 0x7f36500a7320 con 0x7f3650140f70 2024-09-15T02:58:20.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.567+0000 7f364cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:58:20.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.567+0000 7f364cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:58:20.568 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:58:20.569 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":702,"fsid":"7e7cb607-2b58-4a36-a082-99425792b70f","created":"2024-09-15T02:43:09.455959+0000","modified":"2024-09-15T02:58:18.537916+0000","last_up_change":"2024-09-15T02:43:18.346896+0000","last_in_change":"2024-09-15T02:43:11.511047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":97,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:43:19.373532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:43:20.126006+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-15T02:44:19.495432+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"ab035ba6-bb97-46de-9069-4082e1295127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":33156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":33156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":33156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":33156}]},"public_addr":"172.21.15.53:6801/33156","cluster_addr":"172.21.15.53:6804/33156","heartbeat_back_addr":"172.21.15.53:6809/33156","heartbeat_front_addr":"172.21.15.53:6806/33156","state":["exists","up"]},{"osd":1,"uuid":"eb614e79-5e04-45a4-9962-84c7619ddb87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":33157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":33157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":33157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":33157}]},"public_addr":"172.21.15.53:6800/33157","cluster_addr":"172.21.15.53:6803/33157","heartbeat_back_addr":"172.21.15.53:6811/33157","heartbeat_front_addr":"172.21.15.53:6808/33157","state":["exists","up"]},{"osd":2,"uuid":"08ea49a0-28e1-4740-a4b4-d722560b15c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":33158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":33158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":33158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":33158}]},"public_addr":"172.21.15.53:6802/33158","cluster_addr":"172.21.15.53:6805/33158","heartbeat_back_addr":"172.21.15.53:6810/33158","heartbeat_front_addr":"172.21.15.53:6807/33158","state":["exists","up"]},{"osd":3,"uuid":"1a3a89c3-b449-46ba-bc65-3f53371d3af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":33159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":33159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6815","nonce":33159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":33159}]},"public_addr":"172.21.15.53:6812/33159","cluster_addr":"172.21.15.53:6813/33159","heartbeat_back_addr":"172.21.15.53:6815/33159","heartbeat_front_addr":"172.21.15.53:6814/33159","state":["exists","up"]},{"osd":4,"uuid":"d969fe78-c8c8-4c03-a534-2173406b0ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":33348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":33348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6815","nonce":33348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":33348}]},"public_addr":"172.21.15.104:6812/33348","cluster_addr":"172.21.15.104:6813/33348","heartbeat_back_addr":"172.21.15.104:6815/33348","heartbeat_front_addr":"172.21.15.104:6814/33348","state":["exists","up"]},{"osd":5,"uuid":"51e17a2f-42d3-49a6-b71e-dc7e13877c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":33346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":33346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":33346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":33346}]},"public_addr":"172.21.15.104:6800/33346","cluster_addr":"172.21.15.104:6801/33346","heartbeat_back_addr":"172.21.15.104:6803/33346","heartbeat_front_addr":"172.21.15.104:6802/33346","state":["exists","up"]},{"osd":6,"uuid":"c2aae5ac-46f2-4b0d-9e8b-177c43b1c9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":33357}]},"public_addr":"172.21.15.104:6808/33357","cluster_addr":"172.21.15.104:6809/33357","heartbeat_back_addr":"172.21.15.104:6811/33357","heartbeat_front_addr":"172.21.15.104:6810/33357","state":["exists","up"]},{"osd":7,"uuid":"7029aab4-31cc-4219-b7a4-a185564953b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":33347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":33347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":33347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":33347}]},"public_addr":"172.21.15.104:6804/33347","cluster_addr":"172.21.15.104:6805/33347","heartbeat_back_addr":"172.21.15.104:6807/33347","heartbeat_front_addr":"172.21.15.104:6806/33347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.615102+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.701424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.692098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.724924+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.688475+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.598581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.722216+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:43:16.661189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":10,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":13,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":54,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":74,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":81,"snaps":[{"begin":2,"length":1}]},{"pool":90,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":91,"snaps":[{"begin":4,"length":4}]},{"pool":102,"snaps":[{"begin":2,"length":1}]},{"pool":108,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":121,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":128,"snaps":[{"begin":12,"length":1}]},{"pool":150,"snaps":[{"begin":14,"length":1}]},{"pool":162,"snaps":[{"begin":17,"length":1}]},{"pool":166,"snaps":[{"begin":19,"length":2}]},{"pool":179,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 10 librados: watch_flush enter 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 10 librados: watch_flush exit 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 20 client.20333.objecter shutdown clearing up homeless session... 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 10 client.20333.objecter successfully canceled tick 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 >> v2:172.21.15.104:6816/32804 conn(0x7f363006e800 msgr2=0x7f3630070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 --2- 172.21.15.53:0/1930454631 >> v2:172.21.15.104:6816/32804 conn(0x7f363006e800 0x7f3630070cb0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7f3640034000 tx=0x7f3640032000 comp rx=0 tx=0).stop 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 10 monclient: shutdown 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3650140f70 msgr2=0x7f36500a9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3650140f70 0x7f36500a9cf0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f363800c460 tx=0x7f3638007e60 comp rx=0 tx=0).stop 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 shutdown_connections 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3650141340 0x7f365013a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3650140f70 0x7f36500a9cf0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 --2- 172.21.15.53:0/1930454631 >> v2:172.21.15.104:6816/32804 conn(0x7f363006e800 0x7f3630070cb0 unknown :-1 s=CLOSED pgs=3689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 --2- 172.21.15.53:0/1930454631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36500b0a80 0x7f36500aab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:20.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 >> 172.21.15.53:0/1930454631 conn(0x7f3650062d90 msgr2=0x7f36500b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:20.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 shutdown_connections 2024-09-15T02:58:20.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 -- 172.21.15.53:0/1930454631 wait complete. 2024-09-15T02:58:20.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:20.569+0000 7f3656b9d640 1 librados: shutdown 2024-09-15T02:58:21.582 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T02:58:21.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:21.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.694+0000 7f2f77857640 10 monclient: get_monmap_and_config 2024-09-15T02:58:21.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.694+0000 7f2f77857640 10 monclient: build_initial_monmap 2024-09-15T02:58:21.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.694+0000 7f2f77857640 10 monclient: monmap: 2024-09-15T02:58:21.695 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:21.695 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.695 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:21.695135+0000 2024-09-15T02:58:21.695 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:21.695135+0000 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:21.696 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.694+0000 7f2f77857640 1 Processor -- start 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 1 -- start start 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient: init 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2f70140f70 0x7f2f700b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient(hunting): picked mon.noname-c con 0x7f2f70140f70 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f70141340 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient(hunting): picked mon.noname-b con 0x7f2f700b0a80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7013ddd0 con 0x7f2f70141340 2024-09-15T02:58:21.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7013e970 con 0x7f2f70140f70 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700a6c00 con 0x7f2f700b0a80 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.695+0000 7f2f77857640 10 monclient(hunting): authenticate will time out at 1638.265887s 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56230/0 (socket says 172.21.15.53:56230) 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 -- 172.21.15.53:0/13349991 learned_addr learned my addr 172.21.15.53:0/13349991 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): get_auth_request con 0x7f2f700b0a80 auth_method 0 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f755f3640 1 --2- 172.21.15.53:0/13349991 >> v2:172.21.15.104:3300/0 conn(0x7f2f70140f70 0x7f2f700b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f755f3640 10 monclient(hunting): get_auth_request con 0x7f2f70140f70 auth_method 0 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f755f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f755f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f755f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient(hunting): handle_auth_done global_id 20336 payload 274 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 -- 172.21.15.53:0/13349991 >> v2:172.21.15.104:3300/0 conn(0x7f2f70140f70 msgr2=0x7f2f700b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 --2- 172.21.15.53:0/13349991 >> v2:172.21.15.104:3300/0 conn(0x7f2f70140f70 0x7f2f700b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 -- 172.21.15.53:0/13349991 >> v1:172.21.15.104:6789/0 conn(0x7f2f70141340 legacy=0x7f2f7013e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:21.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient: _finish_hunting 0 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 monclient: found mon.noname-b 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.696+0000 7f2f75df4640 1 -- 172.21.15.53:0/13349991 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700a5b30 con 0x7f2f700b0a80 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f75df4640 1 --2- 172.21.15.53:0/13349991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700b0ee0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009f20 tx=0x7f2f68009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d71b636cb1cef1a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 1 -- 172.21.15.53:0/13349991 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68031400 con 0x7f2f700b0a80 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: dump: 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: _finish_auth 0 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 1 -- 172.21.15.53:0/13349991 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6802d050 con 0x7f2f700b0a80 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 5 monclient: authenticate success, global_id 20336 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:21.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 4 set_mon_vals no callback set 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 1 -- 172.21.15.53:0/13349991 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68031790 con 0x7f2f700b0a80 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f67fff640 10 monclient: dump: 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 10 monclient: shutdown 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 1 -- 172.21.15.53:0/13349991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 msgr2=0x7f2f700b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 1 --2- 172.21.15.53:0/13349991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700b0ee0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009f20 tx=0x7f2f68009840 comp rx=0 tx=0).stop 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 1 -- 172.21.15.53:0/13349991 shutdown_connections 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 1 --2- 172.21.15.53:0/13349991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700b0ee0 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 1 --2- 172.21.15.53:0/13349991 >> v2:172.21.15.104:3300/0 conn(0x7f2f70140f70 0x7f2f700b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.697+0000 7f2f77857640 1 -- 172.21.15.53:0/13349991 >> 172.21.15.53:0/13349991 conn(0x7f2f70062d90 msgr2=0x7f2f70063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 -- 172.21.15.53:0/13349991 shutdown_connections 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 -- 172.21.15.53:0/13349991 wait complete. 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 10 monclient: build_initial_monmap 2024-09-15T02:58:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 10 monclient: monmap: 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 librados: starting msgr at 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 librados: starting objecter 2024-09-15T02:58:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 Processor -- start 2024-09-15T02:58:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 -- start start 2024-09-15T02:58:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 librados: setting wanted keys 2024-09-15T02:58:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 1 librados: calling monclient init 2024-09-15T02:58:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.698+0000 7f2f77857640 10 monclient: init 2024-09-15T02:58:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:58:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): picked mon.noname-c con 0x7f2f700b0a80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f70140f70 0x7f2f7013a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): picked mon.noname-b con 0x7f2f70140f70 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f70141340 0x7f2f7013ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): picked mon.noname-a con 0x7f2f70141340 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700ac910 con 0x7f2f70141340 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7013b0b0 con 0x7f2f70140f70 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700ac2b0 con 0x7f2f700b0a80 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f77857640 10 monclient(hunting): authenticate will time out at 1638.269729s 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f75df4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f70141340 0x7f2f7013ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700aa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56240/0 (socket says 172.21.15.53:56240) 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 1 -- 172.21.15.53:0/919094251 learned_addr learned my addr 172.21.15.53:0/919094251 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 10 monclient(hunting): get_auth_request con 0x7f2f700b0a80 auth_method 0 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f75df4640 10 monclient(hunting): get_auth_request con 0x7f2f70141340 auth_method 0 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f75df4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f75df4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f75df4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f74df2640 1 --2- 172.21.15.53:0/919094251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f70140f70 0x7f2f7013a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.699+0000 7f2f755f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f74df2640 10 monclient(hunting): get_auth_request con 0x7f2f70140f70 auth_method 0 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f74df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f74df2640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f74df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f75df4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f75df4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f75df4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f74df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f74df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f74df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 10 monclient(hunting): handle_auth_done global_id 20339 payload 824 2024-09-15T02:58:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 1 -- 172.21.15.53:0/919094251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f70140f70 msgr2=0x7f2f7013a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 1 --2- 172.21.15.53:0/919094251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f70140f70 0x7f2f7013a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 1 -- 172.21.15.53:0/919094251 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f70141340 msgr2=0x7f2f7013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 1 --2- 172.21.15.53:0/919094251 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f70141340 0x7f2f7013ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 10 monclient: _finish_hunting 0 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 1 monclient: found mon.noname-c 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 1 -- 172.21.15.53:0/919094251 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f68009ae0 con 0x7f2f700b0a80 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f755f3640 1 --2- 172.21.15.53:0/919094251 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700aa050 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00d890 tx=0x7f2f6c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 client.?.objecter ms_handle_connect 0x7f2f700b0a80 2024-09-15T02:58:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c00de20 con 0x7f2f700b0a80 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: dump: 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: _finish_auth 0 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c00b870 con 0x7f2f700b0a80 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 5 monclient: authenticate success, global_id 20339 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 10 monclient: _renew_subs 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f700aa620 con 0x7f2f700b0a80 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c00bc10 con 0x7f2f700b0a80 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f657fa640 10 monclient: dump: 2024-09-15T02:58:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f677fe640 4 set_mon_vals no callback set 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 10 client.20339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 10 monclient: _renew_subs 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f700adfe0 con 0x7f2f700b0a80 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.700+0000 7f2f77857640 1 librados: init done 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.701+0000 7f2f77857640 10 client.20339.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.701+0000 7f2f77857640 15 client.20339.objecter _session_command_op_assign -1 1 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.701+0000 7f2f77857640 20 client.20339.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.701+0000 7f2f77857640 10 client.20339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.701+0000 7f2f77857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.701+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f700ab120 con 0x7f2f700b0a80 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f2f6c028430 con 0x7f2f700b0a80 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.104:6816/32804 conn(0x7f2f5406e830 0x7f2f54070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f2f6c08ca10 con 0x7f2f700b0a80 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 10 client.20339.objecter ms_dispatch 0x7f2f70000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 3 client.20339.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 3 client.20339.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 10 client.20339.objecter checking command 1 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.53:6801/33156 conn(0x7f2f54075dc0 0x7f2f54078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 20 client.20339.objecter _get_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 20 client.20339.objecter put_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 15 client.20339.objecter _session_command_op_remove -1 1 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 20 client.20339.objecter _get_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 20 client.20339.objecter get_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 15 client.20339.objecter _session_command_op_assign 0 1 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 20 client.20339.objecter put_session s=0x7f2f54075710 osd=0 4 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 10 client.20339.objecter _send_command 1 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 --> v2:172.21.15.53:6801/33156 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f540788e0 con 0x7f2f54075dc0 2024-09-15T02:58:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 20 client.20339.objecter dump_active .. 0 homeless 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f6c08cde0 con 0x7f2f700b0a80 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.702+0000 7f2f657fa640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f74df2640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.104:6816/32804 conn(0x7f2f5406e830 0x7f2f54070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f74df2640 10 monclient: get_auth_request con 0x7f2f5406e830 auth_method 0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f75df4640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.53:6801/33156 conn(0x7f2f54075dc0 0x7f2f54078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f75df4640 10 monclient: get_auth_request con 0x7f2f54075dc0 auth_method 0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f75df4640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.53:6801/33156 conn(0x7f2f54075dc0 0x7f2f54078250 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f657fa640 10 client.20339.objecter ms_handle_connect 0x7f2f54075dc0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f657fa640 10 client.20339.objecter ms_handle_connect 0x7f2f5406e830 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.703+0000 7f2f74df2640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.104:6816/32804 conn(0x7f2f5406e830 0x7f2f54070ce0 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7f2f700ae410 tx=0x7f2f60006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.704+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== osd.0 v2:172.21.15.53:6801/33156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2f540788e0 con 0x7f2f54075dc0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.704+0000 7f2f657fa640 10 client.20339.objecter ms_dispatch 0x7f2f70000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.704+0000 7f2f657fa640 10 client.20339.objecter _finish_command 1 = system:0 2024-09-15T02:58:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.704+0000 7f2f657fa640 20 client.20339.objecter put_session s=0x7f2f54075710 osd=0 4 2024-09-15T02:58:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.704+0000 7f2f657fa640 15 client.20339.objecter _session_command_op_remove 0 1 2024-09-15T02:58:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 10 client.20339.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 15 client.20339.objecter _session_command_op_assign -1 2 2024-09-15T02:58:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 20 client.20339.objecter _get_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 20 client.20339.objecter put_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 20 client.20339.objecter _get_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 15 client.20339.objecter _session_command_op_remove -1 2 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 20 client.20339.objecter get_session s=0x7f2f54075710 osd=0 3 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 15 client.20339.objecter _session_command_op_assign 0 2 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 20 client.20339.objecter put_session s=0x7f2f54075710 osd=0 4 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 10 client.20339.objecter _send_command 2 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.715+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 --> v2:172.21.15.53:6801/33156 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2f700ae200 con 0x7f2f54075dc0 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f657fa640 1 -- 172.21.15.53:0/919094251 <== osd.0 v2:172.21.15.53:6801/33156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2f700ae200 con 0x7f2f54075dc0 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f657fa640 10 client.20339.objecter ms_dispatch 0x7f2f70000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f657fa640 10 client.20339.objecter _finish_command 2 = system:0 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f657fa640 20 client.20339.objecter put_session s=0x7f2f54075710 osd=0 4 2024-09-15T02:58:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f657fa640 15 client.20339.objecter _session_command_op_remove 0 2 2024-09-15T02:58:21.726 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:21.726 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:21.726 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 10 librados: watch_flush enter 2024-09-15T02:58:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 10 librados: watch_flush exit 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 10 client.20339.objecter close_session for osd.0 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 >> v2:172.21.15.53:6801/33156 conn(0x7f2f54075dc0 msgr2=0x7f2f54078250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.53:6801/33156 conn(0x7f2f54075dc0 0x7f2f54078250 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 20 client.20339.objecter put_session s=0x7f2f54075710 osd=0 1 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 20 client.20339.objecter shutdown clearing up homeless session... 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 10 client.20339.objecter successfully canceled tick 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 >> v2:172.21.15.104:6816/32804 conn(0x7f2f5406e830 msgr2=0x7f2f54070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 --2- 172.21.15.53:0/919094251 >> v2:172.21.15.104:6816/32804 conn(0x7f2f5406e830 0x7f2f54070ce0 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7f2f700ae410 tx=0x7f2f60006f50 comp rx=0 tx=0).stop 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 10 monclient: shutdown 2024-09-15T02:58:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 msgr2=0x7f2f700aa050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 --2- 172.21.15.53:0/919094251 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f700b0a80 0x7f2f700aa050 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00d890 tx=0x7f2f6c008040 comp rx=0 tx=0).stop 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f75df4640 1 -- 172.21.15.53:0/919094251 reap_dead start 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 shutdown_connections 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 >> 172.21.15.53:0/919094251 conn(0x7f2f70062d90 msgr2=0x7f2f700b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 shutdown_connections 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 -- 172.21.15.53:0/919094251 wait complete. 2024-09-15T02:58:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.716+0000 7f2f77857640 1 librados: shutdown 2024-09-15T02:58:21.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T02:58:21.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.826+0000 7fe2c82dd640 10 monclient: get_monmap_and_config 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.826+0000 7fe2c82dd640 10 monclient: build_initial_monmap 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.827+0000 7fe2c82dd640 10 monclient: monmap: 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:21.827772+0000 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:21.827772+0000 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:21.828 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:21.829 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:21.829 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:21.829 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.827+0000 7fe2c82dd640 1 Processor -- start 2024-09-15T02:58:21.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.827+0000 7fe2c82dd640 1 -- start start 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.827+0000 7fe2c82dd640 10 monclient: init 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2c00b0fa0 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2c013b0c0 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c00a5920 0x7fe2c013e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c00a5920 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c00aed10 con 0x7fe2c00b0fa0 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c00b1370 con 0x7fe2c013b0c0 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c6079640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe2c00b0fa0 0x7fe2c013a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59218/0 (socket says 172.21.15.53:59218) 2024-09-15T02:58:21.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c6079640 1 -- 172.21.15.53:0/2765591865 learned_addr learned my addr 172.21.15.53:0/2765591865 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 1 --2- 172.21.15.53:0/2765591865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c00a5920 0x7fe2c013e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 1 -- 172.21.15.53:0/2765591865 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00a6b20 con 0x7fe2c00a5920 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c82dd640 10 monclient(hunting): authenticate will time out at 1638.398686s 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): get_auth_request con 0x7fe2c00a5920 auth_method 0 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c5077640 1 -- 172.21.15.53:0/2765591865 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219646805 0 0) 0x7fe2c00aed10 con 0x7fe2c00b0fa0 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c5077640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c5077640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c5077640 10 monclient(hunting): my global_id is 26247 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c5077640 1 -- 172.21.15.53:0/2765591865 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a80034e0 con 0x7fe2c00b0fa0 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 10 monclient(hunting): handle_auth_done global_id 20342 payload 274 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c5077640 1 -- 172.21.15.53:0/2765591865 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208813428 0 0) 0x7fe2c00b1370 con 0x7fe2c013b0c0 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.828+0000 7fe2c687a640 1 -- 172.21.15.53:0/2765591865 >> v1:172.21.15.104:6789/0 conn(0x7fe2c013b0c0 legacy=0x7fe2c013dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c687a640 1 -- 172.21.15.53:0/2765591865 >> v1:172.21.15.53:6789/0 conn(0x7fe2c00b0fa0 legacy=0x7fe2c013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c687a640 10 monclient: _finish_hunting 0 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c687a640 1 monclient: found mon.noname-b 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c687a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c687a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c687a640 1 -- 172.21.15.53:0/2765591865 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c00a6d20 con 0x7fe2c00a5920 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c687a640 1 --2- 172.21.15.53:0/2765591865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c00a5920 0x7fe2c013e440 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009e90 tx=0x7fe2b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc6fd23470e71347 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 1 -- 172.21.15.53:0/2765591865 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0007550 con 0x7fe2c00a5920 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: dump: 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.840 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: _finish_auth 0 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 1 -- 172.21.15.53:0/2765591865 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b002d050 con 0x7fe2c00a5920 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c82dd640 5 monclient: authenticate success, global_id 20342 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c82dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c82dd640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:21.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c82dd640 4 set_mon_vals no callback set 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 1 -- 172.21.15.53:0/2765591865 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00078e0 con 0x7fe2c00a5920 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.829+0000 7fe2c5077640 10 monclient: dump: 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.842 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 10 monclient: shutdown 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 1 -- 172.21.15.53:0/2765591865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c00a5920 msgr2=0x7fe2c013e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/2765591865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c00a5920 0x7fe2c013e440 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009e90 tx=0x7fe2b0009780 comp rx=0 tx=0).stop 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 1 -- 172.21.15.53:0/2765591865 shutdown_connections 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/2765591865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c00a5920 0x7fe2c013e440 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 1 -- 172.21.15.53:0/2765591865 >> 172.21.15.53:0/2765591865 conn(0x7fe2c0062d90 msgr2=0x7fe2c0063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 1 -- 172.21.15.53:0/2765591865 shutdown_connections 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.830+0000 7fe2c82dd640 1 -- 172.21.15.53:0/2765591865 wait complete. 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 10 monclient: build_initial_monmap 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 10 monclient: monmap: 2024-09-15T02:58:21.843 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 1 librados: starting msgr at 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 1 librados: starting objecter 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 1 Processor -- start 2024-09-15T02:58:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 1 -- start start 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 1 librados: setting wanted keys 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 1 librados: calling monclient init 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.831+0000 7fe2c82dd640 10 monclient: init 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2c00a5920 0x7fe2c013d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c00a5920 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c00b0fa0 0x7fe2c0150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c00b0fa0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c013b0c0 0x7fe2c01508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c013b0c0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00a86f0 con 0x7fe2c00a5920 2024-09-15T02:58:21.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00aa440 con 0x7fe2c00b0fa0 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0150e10 con 0x7fe2c013b0c0 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c82dd640 10 monclient(hunting): authenticate will time out at 1638.402269s 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2c00a5920 0x7fe2c013d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2c00a5920 0x7fe2c013d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52380/0 (socket says 172.21.15.53:52380) 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 -- 172.21.15.53:0/1459549945 learned_addr learned my addr 172.21.15.53:0/1459549945 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c013b0c0 0x7fe2c01508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): get_auth_request con 0x7fe2c00a5920 auth_method 0 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 10 monclient(hunting): get_auth_request con 0x7fe2c013b0c0 auth_method 0 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c00b0fa0 0x7fe2c0150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 10 monclient(hunting): get_auth_request con 0x7fe2c00b0fa0 auth_method 0 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:21.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c5878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient(hunting): handle_auth_done global_id 26250 payload 824 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 -- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c013b0c0 msgr2=0x7fe2c01508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c013b0c0 0x7fe2c01508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 -- 172.21.15.53:0/1459549945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c00b0fa0 msgr2=0x7fe2c0150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c00b0fa0 0x7fe2c0150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient: _finish_hunting 0 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 monclient: found mon.noname-a 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:21.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 -- 172.21.15.53:0/1459549945 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b0009a20 con 0x7fe2c00a5920 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c687a640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c013b0c0 0x7fe2c01508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2c6079640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2c00a5920 0x7fe2c013d250 secure :-1 s=READY pgs=5476 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800c560 tx=0x7fe2b8007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2be7fc640 10 client.?.objecter ms_handle_connect 0x7fe2c00a5920 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2be7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.832+0000 7fe2be7fc640 1 -- 172.21.15.53:0/1459549945 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8005aa0 con 0x7fe2c00a5920 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: dump: 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.849 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: _finish_auth 0 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 5 monclient: authenticate success, global_id 26250 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 1 -- 172.21.15.53:0/1459549945 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b8005c00 con 0x7fe2c00a5920 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 10 monclient: _renew_subs 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c0151090 con 0x7fe2c00a5920 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 1 -- 172.21.15.53:0/1459549945 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80070d0 con 0x7fe2c00a5920 2024-09-15T02:58:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2be7fc640 10 monclient: dump: 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 10 client.26250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:21.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 10 monclient: _renew_subs 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c4876640 4 set_mon_vals no callback set 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c00a9590 con 0x7fe2c00a5920 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.833+0000 7fe2c82dd640 1 librados: init done 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.834+0000 7fe2c82dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.834+0000 7fe2c82dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.834+0000 7fe2c82dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.834+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c00646e0 con 0x7fe2c00a5920 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.834+0000 7fe2be7fc640 1 -- 172.21.15.53:0/1459549945 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7fe2b8007480 con 0x7fe2c00a5920 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.834+0000 7fe2be7fc640 1 --2- 172.21.15.53:0/1459549945 >> v2:172.21.15.104:6816/32804 conn(0x7fe2ac06e830 0x7fe2ac070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2be7fc640 1 -- 172.21.15.53:0/1459549945 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fe2b808aff0 con 0x7fe2c00a5920 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2be7fc640 10 client.26250.objecter ms_dispatch 0x7fe2c0000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2be7fc640 3 client.26250.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:21.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2be7fc640 3 client.26250.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2be7fc640 20 client.26250.objecter dump_active .. 0 homeless 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2c5878640 1 --2- 172.21.15.53:0/1459549945 >> v2:172.21.15.104:6816/32804 conn(0x7fe2ac06e830 0x7fe2ac070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2c5878640 10 monclient: get_auth_request con 0x7fe2ac06e830 auth_method 0 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2c5878640 1 --2- 172.21.15.53:0/1459549945 >> v2:172.21.15.104:6816/32804 conn(0x7fe2ac06e830 0x7fe2ac070ce0 secure :-1 s=READY pgs=3692 cs=0 l=1 rev1=1 crypto rx=0x7fe2c00abba0 tx=0x7fe2b4007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.835+0000 7fe2be7fc640 10 client.26250.objecter ms_handle_connect 0x7fe2ac06e830 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.839+0000 7fe2be7fc640 1 -- 172.21.15.53:0/1459549945 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe2b80551f0 con 0x7fe2c00a5920 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.839+0000 7fe2be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:21.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:21.839+0000 7fe2be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:22.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.002+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2c00aa600 con 0x7fe2ac06e830 2024-09-15T02:58:22.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.003+0000 7fe2be7fc640 1 -- 172.21.15.53:0/1459549945 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe2c00aa600 con 0x7fe2ac06e830 2024-09-15T02:58:22.004 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T02:58:22.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 10 librados: watch_flush enter 2024-09-15T02:58:22.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 10 librados: watch_flush exit 2024-09-15T02:58:22.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 20 client.26250.objecter shutdown clearing up homeless session... 2024-09-15T02:58:22.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 10 client.26250.objecter successfully canceled tick 2024-09-15T02:58:22.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 >> v2:172.21.15.104:6816/32804 conn(0x7fe2ac06e830 msgr2=0x7fe2ac070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/1459549945 >> v2:172.21.15.104:6816/32804 conn(0x7fe2ac06e830 0x7fe2ac070ce0 secure :-1 s=READY pgs=3692 cs=0 l=1 rev1=1 crypto rx=0x7fe2c00abba0 tx=0x7fe2b4007e30 comp rx=0 tx=0).stop 2024-09-15T02:58:22.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 10 monclient: shutdown 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2c00a5920 msgr2=0x7fe2c013d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2c00a5920 0x7fe2c013d250 secure :-1 s=READY pgs=5476 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800c560 tx=0x7fe2b8007d60 comp rx=0 tx=0).stop 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 shutdown_connections 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe2c013b0c0 0x7fe2c01508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/1459549945 >> v2:172.21.15.104:6816/32804 conn(0x7fe2ac06e830 0x7fe2ac070ce0 unknown :-1 s=CLOSED pgs=3692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c00b0fa0 0x7fe2c0150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 --2- 172.21.15.53:0/1459549945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2c00a5920 0x7fe2c013d250 unknown :-1 s=CLOSED pgs=5476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 >> 172.21.15.53:0/1459549945 conn(0x7fe2c0062d90 msgr2=0x7fe2c013dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 shutdown_connections 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 -- 172.21.15.53:0/1459549945 wait complete. 2024-09-15T02:58:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.005+0000 7fe2c82dd640 1 librados: shutdown 2024-09-15T02:58:22.017 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T02:58:22.017 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:22.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.116+0000 7f3b3095d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:22.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.116+0000 7f3b3095d640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.116+0000 7f3b3095d640 10 monclient: monmap: 2024-09-15T02:58:22.117 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.117 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.117 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:22.117197+0000 2024-09-15T02:58:22.117 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:22.117197+0000 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:22.118 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 1 Processor -- start 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 1 -- start start 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 10 monclient: init 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3b2c0ae440 0x7f3b2c13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b2c0ae440 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:22.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b2c06ed40 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b2c138d80 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0ae810 con 0x7f3b2c138d80 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.117+0000 7f3b3095d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0acc00 con 0x7f3b2c06ed40 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b3095d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b3095d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c0acee0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b3095d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b3095d640 10 monclient(hunting): authenticate will time out at 1638.688014s 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2ad9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3b2c138d80 0x7f3b2c145ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59220/0 (socket says 172.21.15.53:59220) 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2ad9d640 1 -- 172.21.15.53:0/618562137 learned_addr learned my addr 172.21.15.53:0/618562137 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 1 --2- 172.21.15.53:0/618562137 >> v2:172.21.15.104:3300/0 conn(0x7f3b2c0ae440 0x7f3b2c13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request con 0x7f3b2c0ae440 auth_method 0 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2959a640 1 -- 172.21.15.53:0/618562137 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244022588 0 0) 0x7f3b2c0ae810 con 0x7f3b2c138d80 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2959a640 10 monclient(hunting): my global_id is 26253 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2959a640 1 -- 172.21.15.53:0/618562137 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b140034e0 con 0x7f3b2c138d80 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2959a640 1 -- 172.21.15.53:0/618562137 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021917261 0 0) 0x7f3b140034e0 con 0x7f3b2c138d80 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_done global_id 24928 payload 274 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 1 -- 172.21.15.53:0/618562137 >> v1:172.21.15.104:6789/0 conn(0x7f3b2c06ed40 legacy=0x7f3b2c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 1 -- 172.21.15.53:0/618562137 >> v1:172.21.15.53:6789/0 conn(0x7f3b2c138d80 legacy=0x7f3b2c145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 1 monclient: found mon.noname-c 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3300/0 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2a59c640 1 -- 172.21.15.53:0/618562137 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0ad1c0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.118+0000 7f3b2959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2a59c640 1 --2- 172.21.15.53:0/618562137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c13ec50 secure :-1 s=READY pgs=5379 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009850 tx=0x7f3b1c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ebd7cc8d9a5271 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 1 -- 172.21.15.53:0/618562137 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b240017f0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: dump: 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.130 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 1 -- 172.21.15.53:0/618562137 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c02aa50 con 0x7f3b2c0ae440 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b3095d640 5 monclient: authenticate success, global_id 24928 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 1 -- 172.21.15.53:0/618562137 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c02add0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b3095d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b3095d640 4 set_mon_vals no callback set 2024-09-15T02:58:22.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.119+0000 7f3b2959a640 10 monclient: dump: 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 10 monclient: shutdown 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 1 -- 172.21.15.53:0/618562137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c0ae440 msgr2=0x7f3b2c13ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 1 --2- 172.21.15.53:0/618562137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c13ec50 secure :-1 s=READY pgs=5379 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009850 tx=0x7f3b1c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 1 -- 172.21.15.53:0/618562137 shutdown_connections 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 1 --2- 172.21.15.53:0/618562137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c13ec50 unknown :-1 s=CLOSED pgs=5379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 1 -- 172.21.15.53:0/618562137 >> 172.21.15.53:0/618562137 conn(0x7f3b2c068590 msgr2=0x7f3b2c068990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 1 -- 172.21.15.53:0/618562137 shutdown_connections 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 1 -- 172.21.15.53:0/618562137 wait complete. 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.120+0000 7f3b3095d640 10 monclient: monmap: 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:22.133 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 librados: starting msgr at 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 librados: starting objecter 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 Processor -- start 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 -- start start 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 librados: setting wanted keys 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 librados: calling monclient init 2024-09-15T02:58:22.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient: init 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c06ed40 0x7f3b2c13da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b2c06ed40 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c06bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b2c0ae440 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b2c138d80 0x7f3b2c150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b2c138d80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c06d440 con 0x7f3b2c0ae440 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c06d720 con 0x7f3b2c06ed40 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c13e7e0 con 0x7f3b2c138d80 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b3095d640 10 monclient(hunting): authenticate will time out at 1638.691885s 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b29d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c06bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.121+0000 7f3b2ad9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b2c138d80 0x7f3b2c150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c06bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52392/0 (socket says 172.21.15.53:52392) 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 -- 172.21.15.53:0/626853390 learned_addr learned my addr 172.21.15.53:0/626853390 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): get_auth_request con 0x7f3b2c0ae440 auth_method 0 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 10 monclient(hunting): get_auth_request con 0x7f3b2c138d80 auth_method 0 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 1 --2- 172.21.15.53:0/626853390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c06ed40 0x7f3b2c13da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request con 0x7f3b2c06ed40 auth_method 0 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient(hunting): handle_auth_done global_id 26256 payload 824 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 -- 172.21.15.53:0/626853390 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b2c138d80 msgr2=0x7f3b2c150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 --2- 172.21.15.53:0/626853390 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b2c138d80 0x7f3b2c150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 -- 172.21.15.53:0/626853390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c06ed40 msgr2=0x7f3b2c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 --2- 172.21.15.53:0/626853390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b2c06ed40 0x7f3b2c13da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 monclient: found mon.noname-a 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 -- 172.21.15.53:0/626853390 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c009390 con 0x7f3b2c0ae440 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b2ad9d640 1 --2- 172.21.15.53:0/626853390 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b2c138d80 0x7f3b2c150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b29d9b640 1 --2- 172.21.15.53:0/626853390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c06bd90 secure :-1 s=READY pgs=5478 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c560 tx=0x7f3b20007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f3b2c0ae440 2024-09-15T02:58:22.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20011590 con 0x7f3b2c0ae440 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: dump: 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 5 monclient: authenticate success, global_id 26256 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 10 monclient: _renew_subs 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b20011710 con 0x7f3b2c0ae440 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c13dfa0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20011ad0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b1a7fc640 10 monclient: dump: 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.140 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b28d99640 4 set_mon_vals no callback set 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 10 client.26256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 10 monclient: _renew_subs 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c0a9a70 con 0x7f3b2c0ae440 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.122+0000 7f3b3095d640 1 librados: init done 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.123+0000 7f3b3095d640 10 client.26256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.123+0000 7f3b3095d640 15 client.26256.objecter _session_command_op_assign -1 1 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.123+0000 7f3b3095d640 20 client.26256.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.123+0000 7f3b3095d640 10 client.26256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.123+0000 7f3b3095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.123+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b2c0646e0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f3b2001b3b0 con 0x7f3b2c0ae440 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.104:6816/32804 conn(0x7f3b0c06e800 0x7f3b0c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f3b2008b020 con 0x7f3b2c0ae440 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 10 client.26256.objecter ms_dispatch 0x7f3b2c000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 3 client.26256.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:22.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 3 client.26256.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 10 client.26256.objecter checking command 1 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.53:6800/33157 conn(0x7f3b0c075d90 0x7f3b0c078220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 20 client.26256.objecter _get_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 20 client.26256.objecter put_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 15 client.26256.objecter _session_command_op_remove -1 1 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 20 client.26256.objecter _get_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 20 client.26256.objecter get_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 15 client.26256.objecter _session_command_op_assign 1 1 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 20 client.26256.objecter put_session s=0x7f3b0c0756e0 osd=1 4 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 10 client.26256.objecter _send_command 1 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 --> v2:172.21.15.53:6800/33157 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b0c0788b0 con 0x7f3b0c075d90 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 20 client.26256.objecter dump_active .. 0 homeless 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b2008d660 con 0x7f3b2c0ae440 2024-09-15T02:58:22.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.124+0000 7f3b1a7fc640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b2a59c640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.104:6816/32804 conn(0x7f3b0c06e800 0x7f3b0c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b2a59c640 10 monclient: get_auth_request con 0x7f3b0c06e800 auth_method 0 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b2ad9d640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.53:6800/33157 conn(0x7f3b0c075d90 0x7f3b0c078220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b2ad9d640 10 monclient: get_auth_request con 0x7f3b0c075d90 auth_method 0 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b2ad9d640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.53:6800/33157 conn(0x7f3b0c075d90 0x7f3b0c078220 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b1a7fc640 10 client.26256.objecter ms_handle_connect 0x7f3b0c075d90 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b1a7fc640 10 client.26256.objecter ms_handle_connect 0x7f3b0c06e800 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b2a59c640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.104:6816/32804 conn(0x7f3b0c06e800 0x7f3b0c070cb0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c034000 tx=0x7f3b1c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== osd.1 v2:172.21.15.53:6800/33157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3b0c0788b0 con 0x7f3b0c075d90 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b1a7fc640 10 client.26256.objecter ms_dispatch 0x7f3b2c000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b1a7fc640 10 client.26256.objecter _finish_command 1 = system:0 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b1a7fc640 20 client.26256.objecter put_session s=0x7f3b0c0756e0 osd=1 4 2024-09-15T02:58:22.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.125+0000 7f3b1a7fc640 15 client.26256.objecter _session_command_op_remove 1 1 2024-09-15T02:58:22.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 10 client.26256.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:22.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 15 client.26256.objecter _session_command_op_assign -1 2 2024-09-15T02:58:22.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 20 client.26256.objecter _get_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 20 client.26256.objecter put_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 20 client.26256.objecter _get_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 15 client.26256.objecter _session_command_op_remove -1 2 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 20 client.26256.objecter get_session s=0x7f3b0c0756e0 osd=1 3 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 15 client.26256.objecter _session_command_op_assign 1 2 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 20 client.26256.objecter put_session s=0x7f3b0c0756e0 osd=1 4 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 10 client.26256.objecter _send_command 2 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 --> v2:172.21.15.53:6800/33157 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3b2c06c2d0 con 0x7f3b0c075d90 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b1a7fc640 1 -- 172.21.15.53:0/626853390 <== osd.1 v2:172.21.15.53:6800/33157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3b2c06c2d0 con 0x7f3b0c075d90 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b1a7fc640 10 client.26256.objecter ms_dispatch 0x7f3b2c000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b1a7fc640 10 client.26256.objecter _finish_command 2 = system:0 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b1a7fc640 20 client.26256.objecter put_session s=0x7f3b0c0756e0 osd=1 4 2024-09-15T02:58:22.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b1a7fc640 15 client.26256.objecter _session_command_op_remove 1 2 2024-09-15T02:58:22.147 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:22.147 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:22.147 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.137+0000 7f3b3095d640 10 librados: watch_flush enter 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 10 librados: watch_flush exit 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 10 client.26256.objecter close_session for osd.1 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 >> v2:172.21.15.53:6800/33157 conn(0x7f3b0c075d90 msgr2=0x7f3b0c078220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.53:6800/33157 conn(0x7f3b0c075d90 0x7f3b0c078220 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 20 client.26256.objecter put_session s=0x7f3b0c0756e0 osd=1 1 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 20 client.26256.objecter shutdown clearing up homeless session... 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 10 client.26256.objecter successfully canceled tick 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 >> v2:172.21.15.104:6816/32804 conn(0x7f3b0c06e800 msgr2=0x7f3b0c070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 --2- 172.21.15.53:0/626853390 >> v2:172.21.15.104:6816/32804 conn(0x7f3b0c06e800 0x7f3b0c070cb0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c034000 tx=0x7f3b1c032000 comp rx=0 tx=0).stop 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 10 monclient: shutdown 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2c0ae440 msgr2=0x7f3b2c06bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 --2- 172.21.15.53:0/626853390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2c0ae440 0x7f3b2c06bd90 secure :-1 s=READY pgs=5478 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c560 tx=0x7f3b20007ef0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b2ad9d640 1 -- 172.21.15.53:0/626853390 reap_dead start 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 shutdown_connections 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 >> 172.21.15.53:0/626853390 conn(0x7f3b2c068590 msgr2=0x7f3b2c136ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 shutdown_connections 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 -- 172.21.15.53:0/626853390 wait complete. 2024-09-15T02:58:22.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.138+0000 7f3b3095d640 1 librados: shutdown 2024-09-15T02:58:22.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T02:58:22.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.246+0000 7f7ae2a57640 10 monclient: get_monmap_and_config 2024-09-15T02:58:22.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.246+0000 7f7ae2a57640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.246+0000 7f7ae2a57640 10 monclient: monmap: 2024-09-15T02:58:22.252 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.252 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.252 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:22.247573+0000 2024-09-15T02:58:22.252 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:22.247573+0000 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.247+0000 7f7ae2a57640 1 Processor -- start 2024-09-15T02:58:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.247+0000 7f7ae2a57640 1 -- start start 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.247+0000 7f7ae2a57640 10 monclient: init 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7adc0b0df0 0x7f7adc0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): picked mon.noname-a con 0x7f7adc0b0df0 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7adc0b11c0 0x7f7adc0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): picked mon.noname-c con 0x7f7adc0b11c0 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0a56d0 0x7f7adc0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): picked mon.noname-b con 0x7f7adc0a56d0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7adc13e920 con 0x7f7adc0b0df0 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7adc0b0710 con 0x7f7adc0b11c0 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc13e400 con 0x7f7adc0a56d0 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae2a57640 10 monclient(hunting): authenticate will time out at 1638.818379s 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0a56d0 0x7f7adc0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7adc0b0df0 0x7f7adc0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0a56d0 0x7f7adc0aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56270/0 (socket says 172.21.15.53:56270) 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 1 -- 172.21.15.53:0/3919094544 learned_addr learned my addr 172.21.15.53:0/3919094544 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 10 monclient(hunting): get_auth_request con 0x7f7adc0a56d0 auth_method 0 2024-09-15T02:58:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): get_auth_request con 0x7f7adc0b0df0 auth_method 0 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 1 --2- 172.21.15.53:0/3919094544 >> v2:172.21.15.104:3300/0 conn(0x7f7adc0b11c0 0x7f7adc0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 10 monclient(hunting): get_auth_request con 0x7f7adc0b11c0 auth_method 0 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient(hunting): handle_auth_done global_id 26259 payload 274 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 1 -- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0a56d0 msgr2=0x7f7adc0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 1 --2- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0a56d0 0x7f7adc0aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 1 -- 172.21.15.53:0/3919094544 >> v2:172.21.15.104:3300/0 conn(0x7f7adc0b11c0 msgr2=0x7f7adc0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 1 --2- 172.21.15.53:0/3919094544 >> v2:172.21.15.104:3300/0 conn(0x7f7adc0b11c0 0x7f7adc0b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 1 monclient: found mon.noname-a 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7adbfff640 1 -- 172.21.15.53:0/3919094544 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc0b16d0 con 0x7f7adc0b0df0 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.248+0000 7f7ae0ff4640 1 --2- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0a56d0 0x7f7adc0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adbfff640 1 --2- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc0afc90 secure :-1 s=READY pgs=5479 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0a6f00 tx=0x7f7acc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e29bc249359bf7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 1 -- 172.21.15.53:0/3919094544 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc02ad00 con 0x7f7adc0b0df0 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: dump: 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.259 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 1 -- 172.21.15.53:0/3919094544 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc02ae60 con 0x7f7adc0b0df0 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7ae2a57640 5 monclient: authenticate success, global_id 26259 2024-09-15T02:58:22.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7ae2a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7ae2a57640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 1 -- 172.21.15.53:0/3919094544 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc032670 con 0x7f7adc0b0df0 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7ae2a57640 4 set_mon_vals no callback set 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.249+0000 7f7adaffd640 10 monclient: dump: 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.261 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 10 monclient: shutdown 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 -- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 msgr2=0x7f7adc0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc0afc90 secure :-1 s=READY pgs=5479 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0a6f00 tx=0x7f7acc0090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 -- 172.21.15.53:0/3919094544 shutdown_connections 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0a56d0 0x7f7adc0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/3919094544 >> v2:172.21.15.104:3300/0 conn(0x7f7adc0b11c0 0x7f7adc0b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/3919094544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc0afc90 unknown :-1 s=CLOSED pgs=5479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 -- 172.21.15.53:0/3919094544 >> 172.21.15.53:0/3919094544 conn(0x7f7adc062d90 msgr2=0x7f7adc063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 -- 172.21.15.53:0/3919094544 shutdown_connections 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 1 -- 172.21.15.53:0/3919094544 wait complete. 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.250+0000 7f7ae2a57640 10 monclient: monmap: 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:22.263 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.251+0000 7f7ae2a57640 1 librados: starting msgr at 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.251+0000 7f7ae2a57640 1 librados: starting objecter 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.251+0000 7f7ae2a57640 1 Processor -- start 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.251+0000 7f7ae2a57640 1 -- start start 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.251+0000 7f7ae2a57640 1 librados: setting wanted keys 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.251+0000 7f7ae2a57640 1 librados: calling monclient init 2024-09-15T02:58:22.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.251+0000 7f7ae2a57640 10 monclient: init 2024-09-15T02:58:22.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:58:22.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0a56d0 0x7f7adc140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): picked mon.noname-b con 0x7f7adc0a56d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:22.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): picked mon.noname-a con 0x7f7adc0b0df0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0b11c0 0x7f7adc14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): picked mon.noname-c con 0x7f7adc0b11c0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc150ee0 con 0x7f7adc0b0df0 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0aa830 con 0x7f7adc0a56d0 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0aaaa0 con 0x7f7adc0b11c0 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient(hunting): authenticate will time out at 1638.822116s 2024-09-15T02:58:22.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0b11c0 0x7f7adc14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52410/0 (socket says 172.21.15.53:52410) 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 10 monclient(hunting): get_auth_request con 0x7f7adc0b11c0 auth_method 0 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 -- 172.21.15.53:0/2931531784 learned_addr learned my addr 172.21.15.53:0/2931531784 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): get_auth_request con 0x7f7adc0b0df0 auth_method 0 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0a56d0 0x7f7adc140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 10 monclient(hunting): get_auth_request con 0x7f7adc0a56d0 auth_method 0 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient(hunting): handle_auth_done global_id 26262 payload 824 2024-09-15T02:58:22.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 -- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0b11c0 msgr2=0x7f7adc14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0b11c0 0x7f7adc14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 -- 172.21.15.53:0/2931531784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0a56d0 msgr2=0x7f7adc140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0a56d0 0x7f7adc140e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 monclient: found mon.noname-a 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 -- 172.21.15.53:0/2931531784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4009ae0 con 0x7f7adc0b0df0 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae0ff4640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0b11c0 0x7f7adc14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7adb7fe640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc150610 secure :-1 s=READY pgs=5480 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000ea50 tx=0x7f7ad00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 client.?.objecter ms_handle_connect 0x7f7adc0b0df0 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:22.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 1 -- 172.21.15.53:0/2931531784 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0009600 con 0x7f7adc0b0df0 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: dump: 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.273 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 1 -- 172.21.15.53:0/2931531784 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad000ca10 con 0x7f7adc0b0df0 2024-09-15T02:58:22.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 5 monclient: authenticate success, global_id 26262 2024-09-15T02:58:22.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient: _renew_subs 2024-09-15T02:58:22.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc009390 con 0x7f7adc0b0df0 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 1 -- 172.21.15.53:0/2931531784 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad000cd90 con 0x7f7adc0b0df0 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ac3fff640 10 monclient: dump: 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.276 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 client.26262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient: _renew_subs 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc14d5c0 con 0x7f7adc0b0df0 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ada7fc640 4 set_mon_vals no callback set 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.252+0000 7f7ae2a57640 1 librados: init done 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.253+0000 7f7ae2a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.253+0000 7f7ae2a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.253+0000 7f7ae2a57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.253+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc151090 con 0x7f7adc0b0df0 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.254+0000 7f7ac3fff640 1 -- 172.21.15.53:0/2931531784 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f7ad0013680 con 0x7f7adc0b0df0 2024-09-15T02:58:22.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.254+0000 7f7ac3fff640 1 --2- 172.21.15.53:0/2931531784 >> v2:172.21.15.104:6816/32804 conn(0x7f7ac806e800 0x7f7ac8070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7ac3fff640 1 -- 172.21.15.53:0/2931531784 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f7ad008bd30 con 0x7f7adc0b0df0 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7ac3fff640 10 client.26262.objecter ms_dispatch 0x7f7adc000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7ac3fff640 3 client.26262.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7ac3fff640 3 client.26262.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7ac3fff640 20 client.26262.objecter dump_active .. 0 homeless 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7adbfff640 1 --2- 172.21.15.53:0/2931531784 >> v2:172.21.15.104:6816/32804 conn(0x7f7ac806e800 0x7f7ac8070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7adbfff640 10 monclient: get_auth_request con 0x7f7ac806e800 auth_method 0 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7adbfff640 1 --2- 172.21.15.53:0/2931531784 >> v2:172.21.15.104:6816/32804 conn(0x7f7ac806e800 0x7f7ac8070cb0 secure :-1 s=READY pgs=3694 cs=0 l=1 rev1=1 crypto rx=0x7f7acc034000 tx=0x7f7acc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.255+0000 7f7ac3fff640 10 client.26262.objecter ms_handle_connect 0x7f7ac806e800 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.259+0000 7f7ac3fff640 1 -- 172.21.15.53:0/2931531784 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7adc151090 con 0x7f7adc0b0df0 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.259+0000 7f7ac3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.259+0000 7f7ac3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:22.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.422+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7adc151090 con 0x7f7ac806e800 2024-09-15T02:58:22.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.423+0000 7f7ac3fff640 1 -- 172.21.15.53:0/2931531784 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7adc151090 con 0x7f7ac806e800 2024-09-15T02:58:22.424 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 10 librados: watch_flush enter 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 10 librados: watch_flush exit 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 20 client.26262.objecter shutdown clearing up homeless session... 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 10 client.26262.objecter successfully canceled tick 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 >> v2:172.21.15.104:6816/32804 conn(0x7f7ac806e800 msgr2=0x7f7ac8070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/2931531784 >> v2:172.21.15.104:6816/32804 conn(0x7f7ac806e800 0x7f7ac8070cb0 secure :-1 s=READY pgs=3694 cs=0 l=1 rev1=1 crypto rx=0x7f7acc034000 tx=0x7f7acc032000 comp rx=0 tx=0).stop 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 10 monclient: shutdown 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 msgr2=0x7f7adc150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc150610 secure :-1 s=READY pgs=5480 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000ea50 tx=0x7f7ad00090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 shutdown_connections 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7adc0b11c0 0x7f7adc14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7adc0b0df0 0x7f7adc150610 unknown :-1 s=CLOSED pgs=5480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/2931531784 >> v2:172.21.15.104:6816/32804 conn(0x7f7ac806e800 0x7f7ac8070cb0 unknown :-1 s=CLOSED pgs=3694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 --2- 172.21.15.53:0/2931531784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0a56d0 0x7f7adc140e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.425+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 >> 172.21.15.53:0/2931531784 conn(0x7f7adc062d90 msgr2=0x7f7adc0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.426+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 shutdown_connections 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.426+0000 7f7ae2a57640 1 -- 172.21.15.53:0/2931531784 wait complete. 2024-09-15T02:58:22.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.426+0000 7f7ae2a57640 1 librados: shutdown 2024-09-15T02:58:22.437 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T02:58:22.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.536+0000 7f84c6cc0640 10 monclient: get_monmap_and_config 2024-09-15T02:58:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.536+0000 7f84c6cc0640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.536+0000 7f84c6cc0640 10 monclient: monmap: 2024-09-15T02:58:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:22.537260+0000 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:22.537260+0000 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:22.538 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 1 Processor -- start 2024-09-15T02:58:22.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 1 -- start start 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient: init 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84c00b0fa0 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c013b0c0 0x7f84c013dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f84c013b0c0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f84c00a5920 0x7f84c013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f84c00a5920 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c00aed10 con 0x7f84c00b0fa0 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c00a6b20 con 0x7f84c00a5920 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c00b1370 con 0x7f84c013b0c0 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.537+0000 7f84c6cc0640 10 monclient(hunting): authenticate will time out at 1639.107999s 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c013b0c0 0x7f84c013dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84c4a5c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f84c00b0fa0 0x7f84c013a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59228/0 (socket says 172.21.15.53:59228) 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84c4a5c640 1 -- 172.21.15.53:0/2135816263 learned_addr learned my addr 172.21.15.53:0/2135816263 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): get_auth_request con 0x7f84c013b0c0 auth_method 0 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84c525d640 1 --2- 172.21.15.53:0/2135816263 >> v2:172.21.15.104:3300/0 conn(0x7f84c00a5920 0x7f84c013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient(hunting): handle_auth_done global_id 20357 payload 274 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 1 -- 172.21.15.53:0/2135816263 >> v2:172.21.15.104:3300/0 conn(0x7f84c00a5920 msgr2=0x7f84c013e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 1 --2- 172.21.15.53:0/2135816263 >> v2:172.21.15.104:3300/0 conn(0x7f84c00a5920 0x7f84c013e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 1 -- 172.21.15.53:0/2135816263 >> v1:172.21.15.53:6789/0 conn(0x7f84c00b0fa0 legacy=0x7f84c013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 1 monclient: found mon.noname-b 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bffff640 1 -- 172.21.15.53:0/2135816263 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c00a6d20 con 0x7f84c013b0c0 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bf7fe640 1 -- 172.21.15.53:0/2135816263 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376429115 0 0) 0x7f84c00aed10 con 0x7f84c00b0fa0 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84c525d640 10 monclient: get_auth_request con 0x7f84c00a5920 auth_method 0 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84c525d640 1 --2- 172.21.15.53:0/2135816263 >> v2:172.21.15.104:3300/0 conn(0x7f84c00a5920 0x7f84c013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.538+0000 7f84bf7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bffff640 1 --2- 172.21.15.53:0/2135816263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c013b0c0 0x7f84c013dd30 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009850 tx=0x7f84ac0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7221848624ae77f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 1 -- 172.21.15.53:0/2135816263 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac02ad00 con 0x7f84c013b0c0 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: dump: 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.551 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 1 -- 172.21.15.53:0/2135816263 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac02ae60 con 0x7f84c013b0c0 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 1 -- 172.21.15.53:0/2135816263 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac032670 con 0x7f84c013b0c0 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 5 monclient: authenticate success, global_id 20357 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:22.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 4 set_mon_vals no callback set 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84bf7fe640 10 monclient: dump: 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.553 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 10 monclient: shutdown 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2135816263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c013b0c0 msgr2=0x7f84c013dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 --2- 172.21.15.53:0/2135816263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c013b0c0 0x7f84c013dd30 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009850 tx=0x7f84ac0090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2135816263 shutdown_connections 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 --2- 172.21.15.53:0/2135816263 >> v2:172.21.15.104:3300/0 conn(0x7f84c00a5920 0x7f84c013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 --2- 172.21.15.53:0/2135816263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c013b0c0 0x7f84c013dd30 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2135816263 >> 172.21.15.53:0/2135816263 conn(0x7f84c0062d90 msgr2=0x7f84c0063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2135816263 shutdown_connections 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.539+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2135816263 wait complete. 2024-09-15T02:58:22.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.540+0000 7f84c6cc0640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.540+0000 7f84c6cc0640 10 monclient: monmap: 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.540+0000 7f84c6cc0640 1 librados: starting msgr at 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.540+0000 7f84c6cc0640 1 librados: starting objecter 2024-09-15T02:58:22.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.540+0000 7f84c6cc0640 1 Processor -- start 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 -- start start 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 librados: setting wanted keys 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 librados: calling monclient init 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient: init 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c00a5920 0x7f84c0150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f84c00a5920 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c00b0fa0 0x7f84c00a8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f84c00b0fa0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c013b0c0 0x7f84c00a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f84c013b0c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:22.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c00a88f0 con 0x7f84c00a5920 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c00a8540 con 0x7f84c013b0c0 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01515a0 con 0x7f84c00b0fa0 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c6cc0640 10 monclient(hunting): authenticate will time out at 1639.111574s 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c00a5920 0x7f84c0150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c00b0fa0 0x7f84c00a8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c00b0fa0 0x7f84c00a8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56306/0 (socket says 172.21.15.53:56306) 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c00a5920 0x7f84c0150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52414/0 (socket says 172.21.15.53:52414) 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 1 -- 172.21.15.53:0/2763147587 learned_addr learned my addr 172.21.15.53:0/2763147587 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): get_auth_request con 0x7f84c00b0fa0 auth_method 0 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 10 monclient(hunting): get_auth_request con 0x7f84c00a5920 auth_method 0 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 1 --2- 172.21.15.53:0/2763147587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c013b0c0 0x7f84c00a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 10 monclient(hunting): get_auth_request con 0x7f84c013b0c0 auth_method 0 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c4a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84c525d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 10 monclient(hunting): handle_auth_done global_id 20360 payload 824 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 1 -- 172.21.15.53:0/2763147587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c013b0c0 msgr2=0x7f84c00a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 1 --2- 172.21.15.53:0/2763147587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c013b0c0 0x7f84c00a8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 1 -- 172.21.15.53:0/2763147587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c00a5920 msgr2=0x7f84c0150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.541+0000 7f84bffff640 1 --2- 172.21.15.53:0/2763147587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c00a5920 0x7f84c0150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bffff640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bffff640 1 monclient: found mon.noname-c 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bffff640 1 -- 172.21.15.53:0/2763147587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac009390 con 0x7f84c00b0fa0 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c4a5c640 1 --2- 172.21.15.53:0/2763147587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c00a5920 0x7f84c0150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bffff640 1 --2- 172.21.15.53:0/2763147587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c00b0fa0 0x7f84c00a8000 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009850 tx=0x7f84ac032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 client.?.objecter ms_handle_connect 0x7f84c00b0fa0 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:22.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac032e20 con 0x7f84c00b0fa0 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: dump: 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.561 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac031a60 con 0x7f84c00b0fa0 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac031e20 con 0x7f84c00b0fa0 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84bcff9640 10 monclient: dump: 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.562 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 5 monclient: authenticate success, global_id 20360 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 10 monclient: _renew_subs 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c0142500 con 0x7f84c00b0fa0 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84beffd640 4 set_mon_vals no callback set 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 10 client.20360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 10 monclient: _renew_subs 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:22.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c00a92c0 con 0x7f84c00b0fa0 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.542+0000 7f84c6cc0640 1 librados: init done 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.543+0000 7f84c6cc0640 10 client.20360.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.543+0000 7f84c6cc0640 15 client.20360.objecter _session_command_op_assign -1 1 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.543+0000 7f84c6cc0640 20 client.20360.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.543+0000 7f84c6cc0640 10 client.20360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.543+0000 7f84c6cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.543+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f84c00a86b0 con 0x7f84c00b0fa0 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.543+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f84ac043030 con 0x7f84c00b0fa0 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.104:6816/32804 conn(0x7f84a406e800 0x7f84a4070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f84ac041070 con 0x7f84c00b0fa0 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 10 client.20360.objecter ms_dispatch 0x7f84c0000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 3 client.20360.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:22.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 3 client.20360.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 10 client.20360.objecter checking command 1 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.53:6802/33158 conn(0x7f84a4075d90 0x7f84a4078220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 20 client.20360.objecter _get_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 20 client.20360.objecter put_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 15 client.20360.objecter _session_command_op_remove -1 1 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 20 client.20360.objecter _get_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 20 client.20360.objecter get_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 15 client.20360.objecter _session_command_op_assign 2 1 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 20 client.20360.objecter put_session s=0x7f84a40756e0 osd=2 4 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 10 client.20360.objecter _send_command 1 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 --> v2:172.21.15.53:6802/33158 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84a40788b0 con 0x7f84a4075d90 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 20 client.20360.objecter dump_active .. 0 homeless 2024-09-15T02:58:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84ac0ae3d0 con 0x7f84c00b0fa0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84bcff9640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84c4a5c640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.104:6816/32804 conn(0x7f84a406e800 0x7f84a4070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84c525d640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.53:6802/33158 conn(0x7f84a4075d90 0x7f84a4078220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84c4a5c640 10 monclient: get_auth_request con 0x7f84a406e800 auth_method 0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.544+0000 7f84c525d640 10 monclient: get_auth_request con 0x7f84a4075d90 auth_method 0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84bcff9640 10 client.20360.objecter ms_handle_connect 0x7f84a4075d90 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84c525d640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.53:6802/33158 conn(0x7f84a4075d90 0x7f84a4078220 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84bcff9640 10 client.20360.objecter ms_handle_connect 0x7f84a406e800 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84c4a5c640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.104:6816/32804 conn(0x7f84a406e800 0x7f84a4070cb0 secure :-1 s=READY pgs=3695 cs=0 l=1 rev1=1 crypto rx=0x7f84b40011e0 tx=0x7f84b4007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== osd.2 v2:172.21.15.53:6802/33158 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f84a40788b0 con 0x7f84a4075d90 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84bcff9640 10 client.20360.objecter ms_dispatch 0x7f84c0000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84bcff9640 10 client.20360.objecter _finish_command 1 = system:0 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84bcff9640 20 client.20360.objecter put_session s=0x7f84a40756e0 osd=2 4 2024-09-15T02:58:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.545+0000 7f84bcff9640 15 client.20360.objecter _session_command_op_remove 2 1 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 10 client.20360.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 15 client.20360.objecter _session_command_op_assign -1 2 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 client.20360.objecter _get_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 client.20360.objecter put_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 client.20360.objecter _get_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 15 client.20360.objecter _session_command_op_remove -1 2 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 client.20360.objecter get_session s=0x7f84a40756e0 osd=2 3 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 15 client.20360.objecter _session_command_op_assign 2 2 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 client.20360.objecter put_session s=0x7f84a40756e0 osd=2 4 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 10 client.20360.objecter _send_command 2 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 --> v2:172.21.15.53:6802/33158 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f84c00646e0 con 0x7f84a4075d90 2024-09-15T02:58:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84bcff9640 1 -- 172.21.15.53:0/2763147587 <== osd.2 v2:172.21.15.53:6802/33158 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f84c00646e0 con 0x7f84a4075d90 2024-09-15T02:58:22.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84bcff9640 10 client.20360.objecter ms_dispatch 0x7f84c0000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:22.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84bcff9640 10 client.20360.objecter _finish_command 2 = system:0 2024-09-15T02:58:22.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84bcff9640 20 client.20360.objecter put_session s=0x7f84a40756e0 osd=2 4 2024-09-15T02:58:22.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84bcff9640 15 client.20360.objecter _session_command_op_remove 2 2 2024-09-15T02:58:22.569 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:22.569 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:22.569 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 10 librados: watch_flush enter 2024-09-15T02:58:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 10 librados: watch_flush exit 2024-09-15T02:58:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 10 client.20360.objecter close_session for osd.2 2024-09-15T02:58:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 >> v2:172.21.15.53:6802/33158 conn(0x7f84a4075d90 msgr2=0x7f84a4078220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.53:6802/33158 conn(0x7f84a4075d90 0x7f84a4078220 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 client.20360.objecter put_session s=0x7f84a40756e0 osd=2 1 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 client.20360.objecter shutdown clearing up homeless session... 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 10 client.20360.objecter successfully canceled tick 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 >> v2:172.21.15.104:6816/32804 conn(0x7f84a406e800 msgr2=0x7f84a4070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 --2- 172.21.15.53:0/2763147587 >> v2:172.21.15.104:6816/32804 conn(0x7f84a406e800 0x7f84a4070cb0 secure :-1 s=READY pgs=3695 cs=0 l=1 rev1=1 crypto rx=0x7f84b40011e0 tx=0x7f84b4007ca0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 10 monclient: shutdown 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c00b0fa0 msgr2=0x7f84c00a8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 --2- 172.21.15.53:0/2763147587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c00b0fa0 0x7f84c00a8000 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009850 tx=0x7f84ac032000 comp rx=0 tx=0).stop 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c525d640 1 -- 172.21.15.53:0/2763147587 reap_dead start 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 shutdown_connections 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 >> 172.21.15.53:0/2763147587 conn(0x7f84c0062d90 msgr2=0x7f84c00aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 shutdown_connections 2024-09-15T02:58:22.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.557+0000 7f84c6cc0640 1 -- 172.21.15.53:0/2763147587 wait complete. 2024-09-15T02:58:22.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.558+0000 7f84c6cc0640 1 librados: shutdown 2024-09-15T02:58:22.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T02:58:22.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.669+0000 7f83e2ac0640 10 monclient: get_monmap_and_config 2024-09-15T02:58:22.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.669+0000 7f83e2ac0640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.669+0000 7f83e2ac0640 10 monclient: monmap: 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:22.670511+0000 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:22.670511+0000 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:22.671 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.670+0000 7f83e2ac0640 1 Processor -- start 2024-09-15T02:58:22.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.670+0000 7f83e2ac0640 1 -- start start 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.670+0000 7f83e2ac0640 10 monclient: init 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83dc0681f0 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83dc0685c0 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f83dc06ec50 0x7f83dc1415a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f83dc06ec50 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83dc069100 con 0x7f83dc0681f0 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83dc06cb70 con 0x7f83dc0685c0 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83dc069450 con 0x7f83dc06ec50 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e2ac0640 10 monclient(hunting): authenticate will time out at 1639.241207s 2024-09-15T02:58:22.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f83dc06ec50 0x7f83dc1415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f83dc06ec50 0x7f83dc1415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52426/0 (socket says 172.21.15.53:52426) 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 1 -- 172.21.15.53:0/2573663808 learned_addr learned my addr 172.21.15.53:0/2573663808 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): get_auth_request con 0x7f83dc06ec50 auth_method 0 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e085c640 1 --1- 172.21.15.53:0/2573663808 >> v1:172.21.15.53:6789/0 conn(0x7f83dc0681f0 0x7f83dc06e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59236/0 (socket says 172.21.15.53:59236) 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient(hunting): handle_auth_done global_id 26271 payload 274 2024-09-15T02:58:22.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 1 -- 172.21.15.53:0/2573663808 >> v1:172.21.15.104:6789/0 conn(0x7f83dc0685c0 legacy=0x7f83dc140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 1 -- 172.21.15.53:0/2573663808 >> v1:172.21.15.53:6789/0 conn(0x7f83dc0681f0 legacy=0x7f83dc06e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 1 monclient: found mon.noname-a 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.671+0000 7f83e105d640 1 -- 172.21.15.53:0/2573663808 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc06cde0 con 0x7f83dc06ec50 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e105d640 1 --2- 172.21.15.53:0/2573663808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc1415a0 secure :-1 s=READY pgs=5483 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f20 tx=0x7f83d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8ce26f829be610 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 1 -- 172.21.15.53:0/2573663808 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4031400 con 0x7f83dc06ec50 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: dump: 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.677 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 1 -- 172.21.15.53:0/2573663808 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d402d050 con 0x7f83dc06ec50 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 5 monclient: authenticate success, global_id 26271 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 1 -- 172.21.15.53:0/2573663808 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d40317d0 con 0x7f83dc06ec50 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 4 set_mon_vals no callback set 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83d37fe640 10 monclient: dump: 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.679 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 10 monclient: shutdown 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2573663808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 msgr2=0x7f83dc1415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.672+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2573663808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc1415a0 secure :-1 s=READY pgs=5483 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f20 tx=0x7f83d4009840 comp rx=0 tx=0).stop 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2573663808 shutdown_connections 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2573663808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc1415a0 unknown :-1 s=CLOSED pgs=5483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2573663808 >> 172.21.15.53:0/2573663808 conn(0x7f83dc063810 msgr2=0x7f83dc0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2573663808 shutdown_connections 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2573663808 wait complete. 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 10 monclient: monmap: 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 librados: starting msgr at 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 librados: starting objecter 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.673+0000 7f83e2ac0640 1 Processor -- start 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 -- start start 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 librados: setting wanted keys 2024-09-15T02:58:22.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 librados: calling monclient init 2024-09-15T02:58:22.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient: init 2024-09-15T02:58:22.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:58:22.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83dc0681f0 0x7f83dc13ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): picked mon.noname-c con 0x7f83dc0681f0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83dc0685c0 0x7f83dc13be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): picked mon.noname-b con 0x7f83dc0685c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f83dc06ec50 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc13d350 con 0x7f83dc06ec50 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc150530 con 0x7f83dc0685c0 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc13b710 con 0x7f83dc0681f0 2024-09-15T02:58:22.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e2ac0640 10 monclient(hunting): authenticate will time out at 1639.244699s 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83dc0681f0 0x7f83dc13ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc150980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52434/0 (socket says 172.21.15.53:52434) 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 1 -- 172.21.15.53:0/2908391857 learned_addr learned my addr 172.21.15.53:0/2908391857 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 10 monclient(hunting): get_auth_request con 0x7f83dc06ec50 auth_method 0 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 10 monclient(hunting): get_auth_request con 0x7f83dc0681f0 auth_method 0 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.674+0000 7f83e085c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 10 monclient(hunting): handle_auth_done global_id 26277 payload 824 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d3fff640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83dc0685c0 0x7f83dc13be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 1 -- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83dc0681f0 msgr2=0x7f83dc13ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83dc0681f0 0x7f83dc13ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 1 -- 172.21.15.53:0/2908391857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83dc0685c0 msgr2=0x7f83dc13be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83dc0685c0 0x7f83dc13be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 1 monclient: found mon.noname-a 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 1 -- 172.21.15.53:0/2908391857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d4009ae0 con 0x7f83dc06ec50 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e085c640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83dc0681f0 0x7f83dc13ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d3fff640 10 monclient: get_auth_request con 0x7f83dc0685c0 auth_method 0 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d3fff640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83dc0685c0 0x7f83dc13be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e105d640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc150980 secure :-1 s=READY pgs=5484 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f80 tx=0x7f83d4036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 client.?.objecter ms_handle_connect 0x7f83dc06ec50 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 1 -- 172.21.15.53:0/2908391857 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4008e50 con 0x7f83dc06ec50 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: dump: 2024-09-15T02:58:22.690 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 1 -- 172.21.15.53:0/2908391857 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d403b410 con 0x7f83dc06ec50 2024-09-15T02:58:22.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 1 -- 172.21.15.53:0/2908391857 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d403b7d0 con 0x7f83dc06ec50 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 5 monclient: authenticate success, global_id 26277 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d0ff9640 10 monclient: dump: 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 10 monclient: _renew_subs 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83d2ffd640 4 set_mon_vals no callback set 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83dc0aa8a0 con 0x7f83dc06ec50 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 10 client.26277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 10 monclient: _renew_subs 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83dc150ec0 con 0x7f83dc06ec50 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.675+0000 7f83e2ac0640 1 librados: init done 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.676+0000 7f83e2ac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.676+0000 7f83e2ac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.676+0000 7f83e2ac0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.676+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83dc1506a0 con 0x7f83dc06ec50 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83d0ff9640 1 -- 172.21.15.53:0/2908391857 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f83d4038050 con 0x7f83dc06ec50 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83d0ff9640 1 --2- 172.21.15.53:0/2908391857 >> v2:172.21.15.104:6816/32804 conn(0x7f83c806e800 0x7f83c8070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83d0ff9640 1 -- 172.21.15.53:0/2908391857 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f83d40ac9b0 con 0x7f83dc06ec50 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83d0ff9640 10 client.26277.objecter ms_dispatch 0x7f83dc000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83d0ff9640 3 client.26277.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83d0ff9640 3 client.26277.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83d0ff9640 20 client.26277.objecter dump_active .. 0 homeless 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83e085c640 1 --2- 172.21.15.53:0/2908391857 >> v2:172.21.15.104:6816/32804 conn(0x7f83c806e800 0x7f83c8070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.677+0000 7f83e085c640 10 monclient: get_auth_request con 0x7f83c806e800 auth_method 0 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.678+0000 7f83d0ff9640 10 client.26277.objecter ms_handle_connect 0x7f83c806e800 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.678+0000 7f83e085c640 1 --2- 172.21.15.53:0/2908391857 >> v2:172.21.15.104:6816/32804 conn(0x7f83c806e800 0x7f83c8070cb0 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 crypto rx=0x7f83d80047a0 tx=0x7f83d8007e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.681+0000 7f83d0ff9640 1 -- 172.21.15.53:0/2908391857 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f83dc1506a0 con 0x7f83dc06ec50 2024-09-15T02:58:22.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.681+0000 7f83d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.681+0000 7f83d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:22.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.843+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f83dc1506a0 con 0x7f83c806e800 2024-09-15T02:58:22.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.844+0000 7f83d0ff9640 1 -- 172.21.15.53:0/2908391857 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83dc1506a0 con 0x7f83c806e800 2024-09-15T02:58:22.845 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T02:58:22.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 10 librados: watch_flush enter 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 10 librados: watch_flush exit 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 20 client.26277.objecter shutdown clearing up homeless session... 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 10 client.26277.objecter successfully canceled tick 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 >> v2:172.21.15.104:6816/32804 conn(0x7f83c806e800 msgr2=0x7f83c8070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2908391857 >> v2:172.21.15.104:6816/32804 conn(0x7f83c806e800 0x7f83c8070cb0 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 crypto rx=0x7f83d80047a0 tx=0x7f83d8007e20 comp rx=0 tx=0).stop 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 10 monclient: shutdown 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 msgr2=0x7f83dc150980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc150980 secure :-1 s=READY pgs=5484 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f80 tx=0x7f83d4036000 comp rx=0 tx=0).stop 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 shutdown_connections 2024-09-15T02:58:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83dc06ec50 0x7f83dc150980 unknown :-1 s=CLOSED pgs=5484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2908391857 >> v2:172.21.15.104:6816/32804 conn(0x7f83c806e800 0x7f83c8070cb0 unknown :-1 s=CLOSED pgs=3696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83dc0685c0 0x7f83dc13be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 --2- 172.21.15.53:0/2908391857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83dc0681f0 0x7f83dc13ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 >> 172.21.15.53:0/2908391857 conn(0x7f83dc063810 msgr2=0x7f83dc0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 shutdown_connections 2024-09-15T02:58:22.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 -- 172.21.15.53:0/2908391857 wait complete. 2024-09-15T02:58:22.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.846+0000 7f83e2ac0640 1 librados: shutdown 2024-09-15T02:58:22.857 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T02:58:22.857 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:22.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.956+0000 7efc59c6f640 10 monclient: get_monmap_and_config 2024-09-15T02:58:22.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.956+0000 7efc59c6f640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.956+0000 7efc59c6f640 10 monclient: monmap: 2024-09-15T02:58:22.957 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:22.957422+0000 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:22.957422+0000 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:22.958 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.957+0000 7efc59c6f640 1 Processor -- start 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.957+0000 7efc59c6f640 1 -- start start 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.957+0000 7efc59c6f640 10 monclient: init 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efc54140f70 0x7efc540b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient(hunting): picked mon.noname-a con 0x7efc54140f70 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc5413e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient(hunting): picked mon.noname-b con 0x7efc54141340 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc540b0a80 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5413ddd0 con 0x7efc540b0a80 2024-09-15T02:58:22.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc540a6c00 con 0x7efc54140f70 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc540a5b30 con 0x7efc54141340 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc59c6f640 10 monclient(hunting): authenticate will time out at 1639.528241s 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc5413e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efc54140f70 0x7efc540b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efc54140f70 0x7efc540b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52446/0 (socket says 172.21.15.53:52446) 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc5413e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56320/0 (socket says 172.21.15.53:56320) 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 -- 172.21.15.53:0/4038736714 learned_addr learned my addr 172.21.15.53:0/4038736714 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): get_auth_request con 0x7efc54140f70 auth_method 0 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 10 monclient(hunting): get_auth_request con 0x7efc54141340 auth_method 0 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient(hunting): handle_auth_done global_id 26280 payload 274 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 -- 172.21.15.53:0/4038736714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 msgr2=0x7efc5413e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 --2- 172.21.15.53:0/4038736714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc5413e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 -- 172.21.15.53:0/4038736714 >> v1:172.21.15.104:6789/0 conn(0x7efc540b0a80 legacy=0x7efc540b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 monclient: found mon.noname-a 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.958+0000 7efc537fe640 1 -- 172.21.15.53:0/4038736714 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5413da60 con 0x7efc54140f70 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc537fe640 1 --2- 172.21.15.53:0/4038736714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc540b00f0 secure :-1 s=READY pgs=5485 cs=0 l=1 rev1=1 crypto rx=0x7efc44009910 tx=0x7efc440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7215afc81339e514 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 1 -- 172.21.15.53:0/4038736714 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4402ad20 con 0x7efc54140f70 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: dump: 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.972 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc59c6f640 5 monclient: authenticate success, global_id 26280 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 1 -- 172.21.15.53:0/4038736714 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4402ae80 con 0x7efc54140f70 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc59c6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc59c6f640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 1 -- 172.21.15.53:0/4038736714 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44032670 con 0x7efc54140f70 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc59c6f640 4 set_mon_vals no callback set 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.959+0000 7efc527fc640 10 monclient: dump: 2024-09-15T02:58:22.973 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 10 monclient: shutdown 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 -- 172.21.15.53:0/4038736714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 msgr2=0x7efc540b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 --2- 172.21.15.53:0/4038736714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc540b00f0 secure :-1 s=READY pgs=5485 cs=0 l=1 rev1=1 crypto rx=0x7efc44009910 tx=0x7efc440091d0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 -- 172.21.15.53:0/4038736714 shutdown_connections 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 --2- 172.21.15.53:0/4038736714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc5413e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 --2- 172.21.15.53:0/4038736714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc540b00f0 unknown :-1 s=CLOSED pgs=5485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 -- 172.21.15.53:0/4038736714 >> 172.21.15.53:0/4038736714 conn(0x7efc54062d90 msgr2=0x7efc54063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 -- 172.21.15.53:0/4038736714 shutdown_connections 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 -- 172.21.15.53:0/4038736714 wait complete. 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 10 monclient: build_initial_monmap 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 10 monclient: monmap: 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:22.975 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 librados: starting msgr at 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.960+0000 7efc59c6f640 1 librados: starting objecter 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 Processor -- start 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 -- start start 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 librados: setting wanted keys 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 librados: calling monclient init 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient: init 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc540b0a80 0x7efc54140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): picked mon.noname-b con 0x7efc540b0a80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc5414deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): picked mon.noname-a con 0x7efc54140f70 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc541512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): picked mon.noname-c con 0x7efc54141340 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5414da20 con 0x7efc54140f70 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540a90f0 con 0x7efc540b0a80 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc540a92b0 con 0x7efc54141340 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:22.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc59c6f640 10 monclient(hunting): authenticate will time out at 1639.531804s 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc52ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc5414deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc52ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc5414deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52450/0 (socket says 172.21.15.53:52450) 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc52ffd640 1 -- 172.21.15.53:0/2143268542 learned_addr learned my addr 172.21.15.53:0/2143268542 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc53fff640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc541512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc52ffd640 10 monclient(hunting): get_auth_request con 0x7efc54140f70 auth_method 0 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc53fff640 10 monclient(hunting): get_auth_request con 0x7efc54141340 auth_method 0 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.961+0000 7efc53fff640 1 -- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 msgr2=0x7efc541512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc53fff640 1 -- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 msgr2=0x7efc541512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc53fff640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc541512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc53fff640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc541512d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc540b0a80 0x7efc54140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 10 monclient(hunting): get_auth_request con 0x7efc540b0a80 auth_method 0 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:22.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 10 monclient(hunting): handle_auth_done global_id 26283 payload 824 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 1 -- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 msgr2=0x7efc541512d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc54141340 0x7efc541512d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 1 -- 172.21.15.53:0/2143268542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc540b0a80 msgr2=0x7efc54140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc540b0a80 0x7efc54140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 1 monclient: found mon.noname-a 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 1 -- 172.21.15.53:0/2143268542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc44009450 con 0x7efc54140f70 2024-09-15T02:58:22.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc52ffd640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc5414deb0 secure :-1 s=READY pgs=5486 cs=0 l=1 rev1=1 crypto rx=0x7efc4800e900 tx=0x7efc480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 client.?.objecter ms_handle_connect 0x7efc54140f70 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4800c8e0 con 0x7efc54140f70 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: dump: 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.981 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 5 monclient: authenticate success, global_id 26283 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4800ca40 con 0x7efc54140f70 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 10 monclient: _renew_subs 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4800cde0 con 0x7efc54140f70 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5414e3f0 con 0x7efc54140f70 2024-09-15T02:58:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc3b7fe640 10 monclient: dump: 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc51ffb640 4 set_mon_vals no callback set 2024-09-15T02:58:22.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 10 client.26283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 10 monclient: _renew_subs 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc54151810 con 0x7efc54140f70 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.962+0000 7efc59c6f640 1 librados: init done 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.963+0000 7efc59c6f640 10 client.26283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.963+0000 7efc59c6f640 15 client.26283.objecter _session_command_op_assign -1 1 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.963+0000 7efc59c6f640 20 client.26283.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.963+0000 7efc59c6f640 10 client.26283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.963+0000 7efc59c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.963+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc540646e0 con 0x7efc54140f70 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7efc48013680 con 0x7efc54140f70 2024-09-15T02:58:22.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.104:6816/32804 conn(0x7efc4006e800 0x7efc40070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7efc4808c080 con 0x7efc54140f70 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 10 client.26283.objecter ms_dispatch 0x7efc54000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 3 client.26283.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 3 client.26283.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 10 client.26283.objecter checking command 1 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.53:6812/33159 conn(0x7efc40075d90 0x7efc40078220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 20 client.26283.objecter _get_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 20 client.26283.objecter put_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 15 client.26283.objecter _session_command_op_remove -1 1 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 20 client.26283.objecter _get_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 20 client.26283.objecter get_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 15 client.26283.objecter _session_command_op_assign 3 1 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 20 client.26283.objecter put_session s=0x7efc400756e0 osd=3 4 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 10 client.26283.objecter _send_command 1 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 --> v2:172.21.15.53:6812/33159 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc400788b0 con 0x7efc40075d90 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.964+0000 7efc3b7fe640 20 client.26283.objecter dump_active .. 0 homeless 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc4808c370 con 0x7efc54140f70 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc3b7fe640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc537fe640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.104:6816/32804 conn(0x7efc4006e800 0x7efc40070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc537fe640 10 monclient: get_auth_request con 0x7efc4006e800 auth_method 0 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc53fff640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.53:6812/33159 conn(0x7efc40075d90 0x7efc40078220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc53fff640 10 monclient: get_auth_request con 0x7efc40075d90 auth_method 0 2024-09-15T02:58:22.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc53fff640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.53:6812/33159 conn(0x7efc40075d90 0x7efc40078220 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc3b7fe640 10 client.26283.objecter ms_handle_connect 0x7efc40075d90 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc3b7fe640 10 client.26283.objecter ms_handle_connect 0x7efc4006e800 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc537fe640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.104:6816/32804 conn(0x7efc4006e800 0x7efc40070cb0 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7efc44034000 tx=0x7efc44032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== osd.3 v2:172.21.15.53:6812/33159 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7efc400788b0 con 0x7efc40075d90 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc3b7fe640 10 client.26283.objecter ms_dispatch 0x7efc54000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.965+0000 7efc3b7fe640 10 client.26283.objecter _finish_command 1 = system:0 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.966+0000 7efc3b7fe640 20 client.26283.objecter put_session s=0x7efc400756e0 osd=3 4 2024-09-15T02:58:22.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.966+0000 7efc3b7fe640 15 client.26283.objecter _session_command_op_remove 3 1 2024-09-15T02:58:22.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 10 client.26283.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:22.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 15 client.26283.objecter _session_command_op_assign -1 2 2024-09-15T02:58:22.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 20 client.26283.objecter _get_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 20 client.26283.objecter put_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 20 client.26283.objecter _get_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 15 client.26283.objecter _session_command_op_remove -1 2 2024-09-15T02:58:22.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 20 client.26283.objecter get_session s=0x7efc400756e0 osd=3 3 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 15 client.26283.objecter _session_command_op_assign 3 2 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 20 client.26283.objecter put_session s=0x7efc400756e0 osd=3 4 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 10 client.26283.objecter _send_command 2 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 --> v2:172.21.15.53:6812/33159 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efc540a5f60 con 0x7efc40075d90 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc3b7fe640 1 -- 172.21.15.53:0/2143268542 <== osd.3 v2:172.21.15.53:6812/33159 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7efc540a5f60 con 0x7efc40075d90 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc3b7fe640 10 client.26283.objecter ms_dispatch 0x7efc54000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc3b7fe640 10 client.26283.objecter _finish_command 2 = system:0 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc3b7fe640 20 client.26283.objecter put_session s=0x7efc400756e0 osd=3 4 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc3b7fe640 15 client.26283.objecter _session_command_op_remove 3 2 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:22.989 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:22.990 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:22.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 10 librados: watch_flush enter 2024-09-15T02:58:22.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 10 librados: watch_flush exit 2024-09-15T02:58:22.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 10 client.26283.objecter close_session for osd.3 2024-09-15T02:58:22.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 >> v2:172.21.15.53:6812/33159 conn(0x7efc40075d90 msgr2=0x7efc40078220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.53:6812/33159 conn(0x7efc40075d90 0x7efc40078220 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 20 client.26283.objecter put_session s=0x7efc400756e0 osd=3 1 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 20 client.26283.objecter shutdown clearing up homeless session... 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.977+0000 7efc59c6f640 10 client.26283.objecter successfully canceled tick 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 >> v2:172.21.15.104:6816/32804 conn(0x7efc4006e800 msgr2=0x7efc40070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 --2- 172.21.15.53:0/2143268542 >> v2:172.21.15.104:6816/32804 conn(0x7efc4006e800 0x7efc40070cb0 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7efc44034000 tx=0x7efc44032000 comp rx=0 tx=0).stop 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 10 monclient: shutdown 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 msgr2=0x7efc5414deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 --2- 172.21.15.53:0/2143268542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc54140f70 0x7efc5414deb0 secure :-1 s=READY pgs=5486 cs=0 l=1 rev1=1 crypto rx=0x7efc4800e900 tx=0x7efc480090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc53fff640 1 -- 172.21.15.53:0/2143268542 reap_dead start 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 shutdown_connections 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 >> 172.21.15.53:0/2143268542 conn(0x7efc54062d90 msgr2=0x7efc540aecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:22.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 shutdown_connections 2024-09-15T02:58:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 -- 172.21.15.53:0/2143268542 wait complete. 2024-09-15T02:58:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:22.978+0000 7efc59c6f640 1 librados: shutdown 2024-09-15T02:58:22.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T02:58:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.090+0000 7f165ecdd640 10 monclient: get_monmap_and_config 2024-09-15T02:58:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.090+0000 7f165ecdd640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.090+0000 7f165ecdd640 10 monclient: monmap: 2024-09-15T02:58:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:23.091214+0000 2024-09-15T02:58:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:23.091214+0000 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:23.092 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 1 Processor -- start 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 1 -- start start 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 10 monclient: init 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16580b0e10 0x7f16580b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 10 monclient(hunting): picked mon.noname-a con 0x7f16580b0e10 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16580b11e0 0x7f1658141400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 10 monclient(hunting): picked mon.noname-c con 0x7f16580b11e0 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16580a5900 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.091+0000 7f165ecdd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1658140f70 con 0x7f16580a5900 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ecdd640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16580b09a0 con 0x7f16580b0e10 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ecdd640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16580aed80 con 0x7f16580b11e0 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ecdd640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ecdd640 10 monclient(hunting): authenticate will time out at 1639.662030s 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16580b0e10 0x7f16580b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16580b0e10 0x7f16580b0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52452/0 (socket says 172.21.15.53:52452) 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 -- 172.21.15.53:0/3319157178 learned_addr learned my addr 172.21.15.53:0/3319157178 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): get_auth_request con 0x7f16580b0e10 auth_method 0 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f1657fff640 1 --2- 172.21.15.53:0/3319157178 >> v2:172.21.15.104:3300/0 conn(0x7f16580b11e0 0x7f1658141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f1657fff640 10 monclient(hunting): get_auth_request con 0x7f16580b11e0 auth_method 0 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f1657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f1657fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f1657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient(hunting): handle_auth_done global_id 26286 payload 274 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 -- 172.21.15.53:0/3319157178 >> v2:172.21.15.104:3300/0 conn(0x7f16580b11e0 msgr2=0x7f1658141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 --2- 172.21.15.53:0/3319157178 >> v2:172.21.15.104:3300/0 conn(0x7f16580b11e0 0x7f1658141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 -- 172.21.15.53:0/3319157178 >> v1:172.21.15.104:6789/0 conn(0x7f16580a5900 legacy=0x7f16580af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 monclient: found mon.noname-a 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.092+0000 7f165ca79640 1 -- 172.21.15.53:0/3319157178 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16580aeef0 con 0x7f16580b0e10 2024-09-15T02:58:23.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ca79640 1 --2- 172.21.15.53:0/3319157178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580b0460 secure :-1 s=READY pgs=5487 cs=0 l=1 rev1=1 crypto rx=0x7f16580711e0 tx=0x7f16480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e271f75bc6890d5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 1 -- 172.21.15.53:0/3319157178 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164802add0 con 0x7f16580b0e10 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: dump: 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.105 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 1 -- 172.21.15.53:0/3319157178 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1648032440 con 0x7f16580b0e10 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 5 monclient: authenticate success, global_id 26286 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 1 -- 172.21.15.53:0/3319157178 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648032820 con 0x7f16580b0e10 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 4 set_mon_vals no callback set 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f16577fe640 10 monclient: dump: 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 10 monclient: shutdown 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 1 -- 172.21.15.53:0/3319157178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 msgr2=0x7f16580b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3319157178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580b0460 secure :-1 s=READY pgs=5487 cs=0 l=1 rev1=1 crypto rx=0x7f16580711e0 tx=0x7f16480091d0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 1 -- 172.21.15.53:0/3319157178 shutdown_connections 2024-09-15T02:58:23.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3319157178 >> v2:172.21.15.104:3300/0 conn(0x7f16580b11e0 0x7f1658141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3319157178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580b0460 unknown :-1 s=CLOSED pgs=5487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.093+0000 7f165ecdd640 1 -- 172.21.15.53:0/3319157178 >> 172.21.15.53:0/3319157178 conn(0x7f1658062d90 msgr2=0x7f1658063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.094+0000 7f165ecdd640 1 -- 172.21.15.53:0/3319157178 shutdown_connections 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.094+0000 7f165ecdd640 1 -- 172.21.15.53:0/3319157178 wait complete. 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.094+0000 7f165ecdd640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.094+0000 7f165ecdd640 10 monclient: monmap: 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.108 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.094+0000 7f165ecdd640 1 librados: starting msgr at 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.094+0000 7f165ecdd640 1 librados: starting objecter 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.094+0000 7f165ecdd640 1 Processor -- start 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 -- start start 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 librados: setting wanted keys 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 librados: calling monclient init 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient: init 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16580a5900 0x7f165814e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): picked mon.noname-c con 0x7f16580a5900 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580a9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): picked mon.noname-a con 0x7f16580b0e10 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16580b11e0 0x7f16580aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): picked mon.noname-b con 0x7f16580b11e0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165814de30 con 0x7f16580b0e10 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16580a9580 con 0x7f16580b11e0 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16580aaa30 con 0x7f16580a5900 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ecdd640 10 monclient(hunting): authenticate will time out at 1639.665755s 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f1657fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580a9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16580a5900 0x7f165814e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f1657fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580a9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52458/0 (socket says 172.21.15.53:52458) 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f1657fff640 1 -- 172.21.15.53:0/3981015227 learned_addr learned my addr 172.21.15.53:0/3981015227 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 10 monclient(hunting): get_auth_request con 0x7f16580a5900 auth_method 0 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f1657fff640 10 monclient(hunting): get_auth_request con 0x7f16580b0e10 auth_method 0 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f1657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f1657fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f1657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165d27a640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16580b11e0 0x7f16580aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165ca79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165d27a640 10 monclient(hunting): get_auth_request con 0x7f16580b11e0 auth_method 0 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.095+0000 7f165d27a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165d27a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165d27a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165d27a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165d27a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165d27a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 10 monclient(hunting): handle_auth_done global_id 20369 payload 824 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 1 -- 172.21.15.53:0/3981015227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16580b11e0 msgr2=0x7f16580aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16580b11e0 0x7f16580aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 1 -- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 msgr2=0x7f16580a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580a9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 1 monclient: found mon.noname-c 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 1 -- 172.21.15.53:0/3981015227 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648009450 con 0x7f16580a5900 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1657fff640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580a9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ca79640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16580a5900 0x7f165814e2c0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f1648031000 tx=0x7f1648032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 client.?.objecter ms_handle_connect 0x7f16580a5900 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:23.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 1 -- 172.21.15.53:0/3981015227 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648000d10 con 0x7f16580a5900 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: dump: 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 5 monclient: authenticate success, global_id 20369 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 1 -- 172.21.15.53:0/3981015227 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1648031de0 con 0x7f16580a5900 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 10 monclient: _renew_subs 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16580a71c0 con 0x7f16580a5900 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 1 -- 172.21.15.53:0/3981015227 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164803d690 con 0x7f16580a5900 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f1654ff9640 10 monclient: dump: 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 10 client.20369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 10 monclient: _renew_subs 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165814e800 con 0x7f16580a5900 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f16567fc640 4 set_mon_vals no callback set 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.096+0000 7f165ecdd640 1 librados: init done 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.097+0000 7f165ecdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.097+0000 7f165ecdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.097+0000 7f165ecdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.097+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16580abd70 con 0x7f16580a5900 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.098+0000 7f1654ff9640 1 -- 172.21.15.53:0/3981015227 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f1648039040 con 0x7f16580a5900 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.098+0000 7f1654ff9640 1 --2- 172.21.15.53:0/3981015227 >> v2:172.21.15.104:6816/32804 conn(0x7f163806e830 0x7f1638070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.098+0000 7f1654ff9640 1 -- 172.21.15.53:0/3981015227 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f16480ab730 con 0x7f16580a5900 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.098+0000 7f1654ff9640 10 client.20369.objecter ms_dispatch 0x7f1658000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.098+0000 7f1654ff9640 3 client.20369.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.098+0000 7f1654ff9640 3 client.20369.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.099+0000 7f1657fff640 1 --2- 172.21.15.53:0/3981015227 >> v2:172.21.15.104:6816/32804 conn(0x7f163806e830 0x7f1638070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.099+0000 7f1657fff640 10 monclient: get_auth_request con 0x7f163806e830 auth_method 0 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.099+0000 7f1654ff9640 20 client.20369.objecter dump_active .. 0 homeless 2024-09-15T02:58:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.099+0000 7f1657fff640 1 --2- 172.21.15.53:0/3981015227 >> v2:172.21.15.104:6816/32804 conn(0x7f163806e830 0x7f1638070ce0 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7f16580b3f90 tx=0x7f1640008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.099+0000 7f1654ff9640 10 client.20369.objecter ms_handle_connect 0x7f163806e830 2024-09-15T02:58:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.103+0000 7f1654ff9640 1 -- 172.21.15.53:0/3981015227 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1648075ae0 con 0x7f16580a5900 2024-09-15T02:58:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.103+0000 7f1654ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.103+0000 7f1654ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:23.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.265+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f16580aabc0 con 0x7f163806e830 2024-09-15T02:58:23.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.265+0000 7f1654ff9640 1 -- 172.21.15.53:0/3981015227 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f16580aabc0 con 0x7f163806e830 2024-09-15T02:58:23.266 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T02:58:23.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 10 librados: watch_flush enter 2024-09-15T02:58:23.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 10 librados: watch_flush exit 2024-09-15T02:58:23.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 20 client.20369.objecter shutdown clearing up homeless session... 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 10 client.20369.objecter successfully canceled tick 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 >> v2:172.21.15.104:6816/32804 conn(0x7f163806e830 msgr2=0x7f1638070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3981015227 >> v2:172.21.15.104:6816/32804 conn(0x7f163806e830 0x7f1638070ce0 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7f16580b3f90 tx=0x7f1640008040 comp rx=0 tx=0).stop 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 10 monclient: shutdown 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16580a5900 msgr2=0x7f165814e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16580a5900 0x7f165814e2c0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f1648031000 tx=0x7f1648032000 comp rx=0 tx=0).stop 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 shutdown_connections 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3981015227 >> v2:172.21.15.104:6816/32804 conn(0x7f163806e830 0x7f1638070ce0 unknown :-1 s=CLOSED pgs=3698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16580b11e0 0x7f16580aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16580b0e10 0x7f16580a9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 --2- 172.21.15.53:0/3981015227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16580a5900 0x7f165814e2c0 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 >> 172.21.15.53:0/3981015227 conn(0x7f1658062d90 msgr2=0x7f16580aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 shutdown_connections 2024-09-15T02:58:23.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 -- 172.21.15.53:0/3981015227 wait complete. 2024-09-15T02:58:23.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.268+0000 7f165ecdd640 1 librados: shutdown 2024-09-15T02:58:23.279 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T02:58:23.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:23.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.378+0000 7f4f8226f640 10 monclient: get_monmap_and_config 2024-09-15T02:58:23.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.378+0000 7f4f8226f640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.378+0000 7f4f8226f640 10 monclient: monmap: 2024-09-15T02:58:23.379 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.379 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.379 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:23.379239+0000 2024-09-15T02:58:23.379 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:23.379239+0000 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:23.380 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 1 Processor -- start 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 1 -- start start 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient: init 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f7c140f70 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f7c141340 0x7f4f7c0b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c141340 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f7c0b0a80 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c13ddd0 con 0x7f4f7c0b0a80 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c0a5b30 con 0x7f4f7c140f70 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7c13e970 con 0x7f4f7c141340 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.379+0000 7f4f8226f640 10 monclient(hunting): authenticate will time out at 1639.949935s 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4f7c0b0a80 0x7f4f7c0b0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59250/0 (socket says 172.21.15.53:59250) 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 1 -- 172.21.15.53:0/2334182662 learned_addr learned my addr 172.21.15.53:0/2334182662 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 1 --2- 172.21.15.53:0/2334182662 >> v2:172.21.15.104:3300/0 conn(0x7f4f7c141340 0x7f4f7c0b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778068991 0 0) 0x7f4f7c13ddd0 con 0x7f4f7c0b0a80 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): my global_id is 26292 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f640034e0 con 0x7f4f7c0b0a80 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f7c141340 auth_method 0 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340601757 0 0) 0x7f4f640034e0 con 0x7f4f7c0b0a80 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 >> v2:172.21.15.104:3300/0 conn(0x7f4f7c141340 msgr2=0x7f4f7c0b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 --2- 172.21.15.53:0/2334182662 >> v2:172.21.15.104:3300/0 conn(0x7f4f7c141340 0x7f4f7c0b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 >> v1:172.21.15.104:6789/0 conn(0x7f4f7c140f70 legacy=0x7f4f7c13e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.53:6789/0 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c13da60 con 0x7f4f7c0b0a80 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8226f640 5 monclient: authenticate success, global_id 26292 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8226f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:23.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 65778851 0 0) 0x7f4f740017f0 con 0x7f4f7c0b0a80 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 1 --2- 172.21.15.53:0/2334182662 >> v2:172.21.15.104:3300/0 conn(0x7f4f7c141340 0x7f4f7c0b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: dump: 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.395 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] but i'm connected to v1:172.21.15.53:6789/0, reconnecting 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 >> v1:172.21.15.53:6789/0 conn(0x7f4f7c0b0a80 legacy=0x7f4f7c0b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f640039d0 0x7f4f64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): picked mon.c con 0x7f4f640039d0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f64004780 0x7f4f64004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): picked mon.b con 0x7f4f64004780 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f64005250 0x7f4f6400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): picked mon.a con 0x7f4f64005250 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c13e970 con 0x7f4f64005250 2024-09-15T02:58:23.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6400c1a0 con 0x7f4f64004780 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6400c360 con 0x7f4f640039d0 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8226f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7bfff640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f640039d0 0x7f4f64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f640039d0 auth_method 0 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f64005250 0x7f4f6400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 10 monclient(hunting): get_auth_request con 0x7f4f64005250 auth_method 0 2024-09-15T02:58:23.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f64004780 0x7f4f64004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f64004780 auth_method 0 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.380+0000 7f4f8080c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:58:23.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 10 monclient(hunting): handle_auth_done global_id 26292 payload 293 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 -- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f640039d0 msgr2=0x7f4f64003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f640039d0 0x7f4f64003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 -- 172.21.15.53:0/2334182662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f64004780 msgr2=0x7f4f64004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f64004780 0x7f4f64004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 monclient: found mon.a 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 -- 172.21.15.53:0/2334182662 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640034e0 con 0x7f4f64005250 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f7bfff640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f640039d0 0x7f4f64003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 -- 172.21.15.53:0/2334182662 reap_dead start 2024-09-15T02:58:23.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.381+0000 7f4f8080c640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f64005250 0x7f4f6400b9d0 secure :-1 s=READY pgs=5489 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400a930 tx=0x7f4f740021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ab8a169342cd5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7402c4e0 con 0x7f4f64005250 2024-09-15T02:58:23.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 10 monclient: dump: 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7402c660 con 0x7f4f64005250 2024-09-15T02:58:23.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 1 -- 172.21.15.53:0/2334182662 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7402ca20 con 0x7f4f64005250 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 4 set_mon_vals no callback set 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f7affd640 10 monclient: dump: 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 10 monclient: shutdown 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 1 -- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f64005250 msgr2=0x7f4f6400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f64005250 0x7f4f6400b9d0 secure :-1 s=READY pgs=5489 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400a930 tx=0x7f4f740021d0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 1 -- 172.21.15.53:0/2334182662 shutdown_connections 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 1 --2- 172.21.15.53:0/2334182662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f64005250 0x7f4f6400b9d0 unknown :-1 s=CLOSED pgs=5489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 1 -- 172.21.15.53:0/2334182662 >> 172.21.15.53:0/2334182662 conn(0x7f4f7c062d90 msgr2=0x7f4f7c063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 1 -- 172.21.15.53:0/2334182662 shutdown_connections 2024-09-15T02:58:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.382+0000 7f4f8226f640 1 -- 172.21.15.53:0/2334182662 wait complete. 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 10 monclient: monmap: 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 1 librados: starting msgr at 2024-09-15T02:58:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 1 librados: starting objecter 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 1 Processor -- start 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 1 -- start start 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 1 librados: setting wanted keys 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 1 librados: calling monclient init 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.383+0000 7f4f8226f640 10 monclient: init 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f64005250 0x7f4f7c0aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f64005250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0a9ff0 0x7f4f7c13a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f7c0a9ff0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f7c0aa440 0x7f4f7c13aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c0aa440 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c0a77b0 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c13b020 con 0x7f4f7c0aa440 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c0ac1b0 con 0x7f4f64005250 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8226f640 10 monclient(hunting): authenticate will time out at 1639.954384s 2024-09-15T02:58:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f64005250 0x7f4f7c0aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0a9ff0 0x7f4f7c13a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0a9ff0 0x7f4f7c13a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52482/0 (socket says 172.21.15.53:52482) 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f64005250 0x7f4f7c0aae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56368/0 (socket says 172.21.15.53:56368) 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 -- 172.21.15.53:0/2299765587 learned_addr learned my addr 172.21.15.53:0/2299765587 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f7c0a9ff0 auth_method 0 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f64005250 auth_method 0 2024-09-15T02:58:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 1 --2- 172.21.15.53:0/2299765587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f7c0aa440 0x7f4f7c13aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 10 monclient(hunting): get_auth_request con 0x7f4f7c0aa440 auth_method 0 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f8080c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_done global_id 26295 payload 824 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 -- 172.21.15.53:0/2299765587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f64005250 msgr2=0x7f4f7c0aae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 --2- 172.21.15.53:0/2299765587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f64005250 0x7f4f7c0aae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 -- 172.21.15.53:0/2299765587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f7c0aa440 msgr2=0x7f4f7c13aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 --2- 172.21.15.53:0/2299765587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f7c0aa440 0x7f4f7c13aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 monclient: found mon.noname-a 2024-09-15T02:58:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 -- 172.21.15.53:0/2299765587 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0a7c00 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.384+0000 7f4f7b7fe640 1 --2- 172.21.15.53:0/2299765587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0a9ff0 0x7f4f7c13a5a0 secure :-1 s=READY pgs=5490 cs=0 l=1 rev1=1 crypto rx=0x7f4f70010aa0 tx=0x7f4f70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 client.?.objecter ms_handle_connect 0x7f4f7c0a9ff0 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000ea90 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: dump: 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7000ebf0 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 5 monclient: authenticate success, global_id 26295 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 monclient: _renew_subs 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c0a7e50 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70014430 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f63fff640 10 monclient: dump: 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 client.26295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 monclient: _renew_subs 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c0a8370 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 1 librados: init done 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f7a7fc640 4 set_mon_vals no callback set 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 client.26295.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 15 client.26295.objecter _session_command_op_assign -1 1 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 20 client.26295.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 client.26295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.385+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f7c0a7580 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f4f70014820 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6816/32804 conn(0x7f4f6806e800 0x7f4f68070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f4f7008c6d0 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 10 client.26295.objecter ms_dispatch 0x7f4f7c000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 3 client.26295.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 3 client.26295.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 10 client.26295.objecter checking command 1 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6812/33348 conn(0x7f4f68075d90 0x7f4f68078220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 20 client.26295.objecter _get_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 20 client.26295.objecter put_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 15 client.26295.objecter _session_command_op_remove -1 1 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 20 client.26295.objecter _get_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 20 client.26295.objecter get_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 15 client.26295.objecter _session_command_op_assign 4 1 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 20 client.26295.objecter put_session s=0x7f4f680756e0 osd=4 4 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 10 client.26295.objecter _send_command 1 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 --> v2:172.21.15.104:6812/33348 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f680788b0 con 0x7f4f68075d90 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 20 client.26295.objecter dump_active .. 0 homeless 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f7008caa0 con 0x7f4f7c0a9ff0 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f63fff640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f7bfff640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6816/32804 conn(0x7f4f6806e800 0x7f4f68070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.387+0000 7f4f7bfff640 10 monclient: get_auth_request con 0x7f4f6806e800 auth_method 0 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.388+0000 7f4f8080c640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6812/33348 conn(0x7f4f68075d90 0x7f4f68078220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.388+0000 7f4f8080c640 10 monclient: get_auth_request con 0x7f4f68075d90 auth_method 0 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.388+0000 7f4f63fff640 10 client.26295.objecter ms_handle_connect 0x7f4f6806e800 2024-09-15T02:58:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.388+0000 7f4f7bfff640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6816/32804 conn(0x7f4f6806e800 0x7f4f68070cb0 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0012f0 tx=0x7f4f6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.388+0000 7f4f63fff640 10 client.26295.objecter ms_handle_connect 0x7f4f68075d90 2024-09-15T02:58:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.388+0000 7f4f8080c640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6812/33348 conn(0x7f4f68075d90 0x7f4f68078220 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.389+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== osd.4 v2:172.21.15.104:6812/33348 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4f680788b0 con 0x7f4f68075d90 2024-09-15T02:58:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.389+0000 7f4f63fff640 10 client.26295.objecter ms_dispatch 0x7f4f7c000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.389+0000 7f4f63fff640 10 client.26295.objecter _finish_command 1 = system:0 2024-09-15T02:58:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.389+0000 7f4f63fff640 20 client.26295.objecter put_session s=0x7f4f680756e0 osd=4 4 2024-09-15T02:58:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.389+0000 7f4f63fff640 15 client.26295.objecter _session_command_op_remove 4 1 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 10 client.26295.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 15 client.26295.objecter _session_command_op_assign -1 2 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 20 client.26295.objecter _get_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 20 client.26295.objecter put_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 20 client.26295.objecter _get_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 15 client.26295.objecter _session_command_op_remove -1 2 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 20 client.26295.objecter get_session s=0x7f4f680756e0 osd=4 3 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 15 client.26295.objecter _session_command_op_assign 4 2 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 20 client.26295.objecter put_session s=0x7f4f680756e0 osd=4 4 2024-09-15T02:58:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 10 client.26295.objecter _send_command 2 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.400+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 --> v2:172.21.15.104:6812/33348 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4f7c0a7fe0 con 0x7f4f68075d90 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f63fff640 1 -- 172.21.15.53:0/2299765587 <== osd.4 v2:172.21.15.104:6812/33348 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4f7c0a7fe0 con 0x7f4f68075d90 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f63fff640 10 client.26295.objecter ms_dispatch 0x7f4f7c000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f63fff640 10 client.26295.objecter _finish_command 2 = system:0 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f63fff640 20 client.26295.objecter put_session s=0x7f4f680756e0 osd=4 4 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f63fff640 15 client.26295.objecter _session_command_op_remove 4 2 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:23.419 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:23.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 10 librados: watch_flush enter 2024-09-15T02:58:23.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 10 librados: watch_flush exit 2024-09-15T02:58:23.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 10 client.26295.objecter close_session for osd.4 2024-09-15T02:58:23.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6812/33348 conn(0x7f4f68075d90 msgr2=0x7f4f68078220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6812/33348 conn(0x7f4f68075d90 0x7f4f68078220 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 20 client.26295.objecter put_session s=0x7f4f680756e0 osd=4 1 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 20 client.26295.objecter shutdown clearing up homeless session... 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 10 client.26295.objecter successfully canceled tick 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6816/32804 conn(0x7f4f6806e800 msgr2=0x7f4f68070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 --2- 172.21.15.53:0/2299765587 >> v2:172.21.15.104:6816/32804 conn(0x7f4f6806e800 0x7f4f68070cb0 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0012f0 tx=0x7f4f6c009040 comp rx=0 tx=0).stop 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 10 monclient: shutdown 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0a9ff0 msgr2=0x7f4f7c13a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 --2- 172.21.15.53:0/2299765587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0a9ff0 0x7f4f7c13a5a0 secure :-1 s=READY pgs=5490 cs=0 l=1 rev1=1 crypto rx=0x7f4f70010aa0 tx=0x7f4f70008040 comp rx=0 tx=0).stop 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8080c640 1 -- 172.21.15.53:0/2299765587 reap_dead start 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 shutdown_connections 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 >> 172.21.15.53:0/2299765587 conn(0x7f4f7c062d90 msgr2=0x7f4f7c0b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 shutdown_connections 2024-09-15T02:58:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 -- 172.21.15.53:0/2299765587 wait complete. 2024-09-15T02:58:23.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.401+0000 7f4f8226f640 1 librados: shutdown 2024-09-15T02:58:23.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.520+0000 7f1a0ad5d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.520+0000 7f1a0ad5d640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.520+0000 7f1a0ad5d640 10 monclient: monmap: 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:23.521032+0000 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:23.521032+0000 2024-09-15T02:58:23.521 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.522 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.522 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:23.522 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:23.522 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:23.522 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:23.522 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:23.522 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 1 Processor -- start 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 1 -- start start 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient: init 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a040b0e80 0x7f1a040b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a040b0e80 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a040b1250 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a040af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a0413e600 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0413da60 con 0x7f1a040b1250 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a040b0990 con 0x7f1a040b0e80 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040a6a30 con 0x7f1a0413e600 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a0ad5d640 10 monclient(hunting): authenticate will time out at 1640.091796s 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a092fa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a040af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a08af9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a040b0e80 0x7f1a040b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a092fa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a040af3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56372/0 (socket says 172.21.15.53:56372) 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a03fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1a040b1250 0x7f1a0413def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59252/0 (socket says 172.21.15.53:59252) 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a092fa640 1 -- 172.21.15.53:0/1620019077 learned_addr learned my addr 172.21.15.53:0/1620019077 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a08af9640 1 --2- 172.21.15.53:0/1620019077 >> v2:172.21.15.53:3300/0 conn(0x7f1a040b0e80 0x7f1a040b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52488/0 (socket says 172.21.15.53:52488) 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a092fa640 10 monclient(hunting): get_auth_request con 0x7f1a0413e600 auth_method 0 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a092fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a092fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a092fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a08af9640 10 monclient(hunting): get_auth_request con 0x7f1a040b0e80 auth_method 0 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a08af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.521+0000 7f1a08af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a092fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a092fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a092fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 10 monclient(hunting): handle_auth_done global_id 26298 payload 274 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 1 -- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 msgr2=0x7f1a040af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 1 --2- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a040af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 1 -- 172.21.15.53:0/1620019077 >> v1:172.21.15.53:6789/0 conn(0x7f1a040b1250 legacy=0x7f1a0413def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 1 monclient: found mon.noname-a 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a08af9640 1 -- 172.21.15.53:0/1620019077 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a040a5a00 con 0x7f1a040b0e80 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a037fe640 1 -- 172.21.15.53:0/1620019077 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366298402 0 0) 0x7f1a0413da60 con 0x7f1a040b1250 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a037fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.522+0000 7f1a092fa640 1 --2- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a040af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a08af9640 1 --2- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040b0450 secure :-1 s=READY pgs=5492 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009850 tx=0x7f19f80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cd4656ce59efb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 1 -- 172.21.15.53:0/1620019077 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f802ad00 con 0x7f1a040b0e80 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:23.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: dump: 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 5 monclient: authenticate success, global_id 26298 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 1 -- 172.21.15.53:0/1620019077 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f802ae60 con 0x7f1a040b0e80 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 4 set_mon_vals no callback set 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 1 -- 172.21.15.53:0/1620019077 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8032670 con 0x7f1a040b0e80 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a037fe640 10 monclient: dump: 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.535 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 10 monclient: shutdown 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 msgr2=0x7f1a040b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040b0450 secure :-1 s=READY pgs=5492 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009850 tx=0x7f19f80090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/1620019077 shutdown_connections 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a040af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/1620019077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040b0450 unknown :-1 s=CLOSED pgs=5492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/1620019077 >> 172.21.15.53:0/1620019077 conn(0x7f1a04062d90 msgr2=0x7f1a04063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/1620019077 shutdown_connections 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.523+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/1620019077 wait complete. 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 10 monclient: monmap: 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:23.537 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:23.538 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 1 librados: starting msgr at 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 1 librados: starting objecter 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 1 Processor -- start 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 1 -- start start 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 1 librados: setting wanted keys 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 1 librados: calling monclient init 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.524+0000 7f1a0ad5d640 10 monclient: init 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a040b0e80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a040b1250 0x7f1a040a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a040b1250 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:23.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a0413a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a0413e600 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040ae2b0 con 0x7f1a040b0e80 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040a7340 con 0x7f1a040b1250 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040aaef0 con 0x7f1a0413e600 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a0ad5d640 10 monclient(hunting): authenticate will time out at 1640.095477s 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040aa9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52492/0 (socket says 172.21.15.53:52492) 2024-09-15T02:58:23.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a0413a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 -- 172.21.15.53:0/4043883562 learned_addr learned my addr 172.21.15.53:0/4043883562 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): get_auth_request con 0x7f1a040b0e80 auth_method 0 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 10 monclient(hunting): get_auth_request con 0x7f1a0413e600 auth_method 0 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a03fff640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a040b1250 0x7f1a040a9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient(hunting): handle_auth_done global_id 26304 payload 824 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 -- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 msgr2=0x7f1a0413a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a0413a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 -- 172.21.15.53:0/4043883562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a040b1250 msgr2=0x7f1a040a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a040b1250 0x7f1a040a9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 monclient: found mon.noname-a 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a08af9640 1 -- 172.21.15.53:0/4043883562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc009ae0 con 0x7f1a040b0e80 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a03fff640 10 monclient: get_auth_request con 0x7f1a040b1250 auth_method 0 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a092fa640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a0413a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.525+0000 7f1a03fff640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a040b1250 0x7f1a040a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a08af9640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040aa9b0 secure :-1 s=READY pgs=5493 cs=0 l=1 rev1=1 crypto rx=0x7f19f80098b0 tx=0x7f19f8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 client.?.objecter ms_handle_connect 0x7f1a040b0e80 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 1 -- 172.21.15.53:0/4043883562 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f80316b0 con 0x7f1a040b0e80 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: dump: 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.543 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 1 -- 172.21.15.53:0/4043883562 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f8031810 con 0x7f1a040b0e80 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 5 monclient: authenticate success, global_id 26304 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 10 monclient: _renew_subs 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f8009390 con 0x7f1a040b0e80 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 1 -- 172.21.15.53:0/4043883562 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8031bd0 con 0x7f1a040b0e80 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a00ff9640 10 monclient: dump: 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.545 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a02ffd640 4 set_mon_vals no callback set 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 10 client.26304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 10 monclient: _renew_subs 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0413a710 con 0x7f1a040b0e80 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.526+0000 7f1a0ad5d640 1 librados: init done 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.527+0000 7f1a0ad5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.527+0000 7f1a0ad5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.527+0000 7f1a0ad5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.527+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a04063ff0 con 0x7f1a040b0e80 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.527+0000 7f1a00ff9640 1 -- 172.21.15.53:0/4043883562 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f19f8043030 con 0x7f1a040b0e80 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a00ff9640 1 --2- 172.21.15.53:0/4043883562 >> v2:172.21.15.104:6816/32804 conn(0x7f19e806e800 0x7f19e8070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a00ff9640 1 -- 172.21.15.53:0/4043883562 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f19f8041070 con 0x7f1a040b0e80 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a00ff9640 10 client.26304.objecter ms_dispatch 0x7f1a04000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a00ff9640 3 client.26304.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a00ff9640 3 client.26304.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a03fff640 1 --2- 172.21.15.53:0/4043883562 >> v2:172.21.15.104:6816/32804 conn(0x7f19e806e800 0x7f19e8070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a03fff640 10 monclient: get_auth_request con 0x7f19e806e800 auth_method 0 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.528+0000 7f1a00ff9640 20 client.26304.objecter dump_active .. 0 homeless 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.529+0000 7f1a03fff640 1 --2- 172.21.15.53:0/4043883562 >> v2:172.21.15.104:6816/32804 conn(0x7f19e806e800 0x7f19e8070cb0 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 crypto rx=0x7f1a040abc40 tx=0x7f19f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.529+0000 7f1a00ff9640 10 client.26304.objecter ms_handle_connect 0x7f19e806e800 2024-09-15T02:58:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.532+0000 7f1a00ff9640 1 -- 172.21.15.53:0/4043883562 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f19f80762e0 con 0x7f1a040b0e80 2024-09-15T02:58:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.532+0000 7f1a00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.532+0000 7f1a00ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:23.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.698+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a040a66e0 con 0x7f19e806e800 2024-09-15T02:58:23.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.699+0000 7f1a00ff9640 1 -- 172.21.15.53:0/4043883562 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1a040a66e0 con 0x7f19e806e800 2024-09-15T02:58:23.700 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 10 librados: watch_flush enter 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 10 librados: watch_flush exit 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 20 client.26304.objecter shutdown clearing up homeless session... 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 10 client.26304.objecter successfully canceled tick 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 >> v2:172.21.15.104:6816/32804 conn(0x7f19e806e800 msgr2=0x7f19e8070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/4043883562 >> v2:172.21.15.104:6816/32804 conn(0x7f19e806e800 0x7f19e8070cb0 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 crypto rx=0x7f1a040abc40 tx=0x7f19f0009040 comp rx=0 tx=0).stop 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 10 monclient: shutdown 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 msgr2=0x7f1a040aa9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040aa9b0 secure :-1 s=READY pgs=5493 cs=0 l=1 rev1=1 crypto rx=0x7f19f80098b0 tx=0x7f19f8032000 comp rx=0 tx=0).stop 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 shutdown_connections 2024-09-15T02:58:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a0413e600 0x7f1a0413a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/4043883562 >> v2:172.21.15.104:6816/32804 conn(0x7f19e806e800 0x7f19e8070cb0 unknown :-1 s=CLOSED pgs=3700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a040b1250 0x7f1a040a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 --2- 172.21.15.53:0/4043883562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a040b0e80 0x7f1a040aa9b0 unknown :-1 s=CLOSED pgs=5493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 >> 172.21.15.53:0/4043883562 conn(0x7f1a04062d90 msgr2=0x7f1a0413c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 shutdown_connections 2024-09-15T02:58:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 -- 172.21.15.53:0/4043883562 wait complete. 2024-09-15T02:58:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.701+0000 7f1a0ad5d640 1 librados: shutdown 2024-09-15T02:58:23.712 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T02:58:23.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:23.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.811+0000 7f7fcaaaf640 10 monclient: get_monmap_and_config 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.811+0000 7f7fcaaaf640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.811+0000 7f7fcaaaf640 10 monclient: monmap: 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:23.812674+0000 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:23.812674+0000 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:23.813 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.812+0000 7f7fcaaaf640 1 Processor -- start 2024-09-15T02:58:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.812+0000 7f7fcaaaf640 1 -- start start 2024-09-15T02:58:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.812+0000 7f7fcaaaf640 10 monclient: init 2024-09-15T02:58:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T02:58:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7fc40b0e20 0x7f7fc40b0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc40b0e20 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7fc40b11f0 0x7f7fc413de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc40b11f0 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7fc40a5900 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc413d9f0 con 0x7f7fc40a5900 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc40aedf0 con 0x7f7fc40b11f0 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc40aef30 con 0x7f7fc40b0e20 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fcaaaf640 10 monclient(hunting): authenticate will time out at 1640.383355s 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7fc40b11f0 0x7f7fc413de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc904c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7fc40a5900 0x7f7fc40af450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59264/0 (socket says 172.21.15.53:59264) 2024-09-15T02:58:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7fc40b11f0 0x7f7fc413de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52496/0 (socket says 172.21.15.53:52496) 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc904c640 1 -- 172.21.15.53:0/309014219 learned_addr learned my addr 172.21.15.53:0/309014219 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): get_auth_request con 0x7f7fc40b11f0 auth_method 0 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc884b640 1 --2- 172.21.15.53:0/309014219 >> v2:172.21.15.104:3300/0 conn(0x7f7fc40b0e20 0x7f7fc40b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient(hunting): handle_auth_done global_id 26307 payload 274 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 1 -- 172.21.15.53:0/309014219 >> v2:172.21.15.104:3300/0 conn(0x7f7fc40b0e20 msgr2=0x7f7fc40b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 1 --2- 172.21.15.53:0/309014219 >> v2:172.21.15.104:3300/0 conn(0x7f7fc40b0e20 0x7f7fc40b0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 1 -- 172.21.15.53:0/309014219 >> v1:172.21.15.53:6789/0 conn(0x7f7fc40a5900 legacy=0x7f7fc40af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 1 monclient: found mon.noname-a 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.813+0000 7f7fc3fff640 1 -- 172.21.15.53:0/309014219 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc40afb60 con 0x7f7fc40b11f0 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc3fff640 1 --2- 172.21.15.53:0/309014219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40b11f0 0x7f7fc413de80 secure :-1 s=READY pgs=5495 cs=0 l=1 rev1=1 crypto rx=0x7f7fc406f590 tx=0x7f7fb80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d824813728c991f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 1 -- 172.21.15.53:0/309014219 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb802ad20 con 0x7f7fc40b11f0 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: dump: 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 1 -- 172.21.15.53:0/309014219 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb802ae80 con 0x7f7fc40b11f0 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fcaaaf640 5 monclient: authenticate success, global_id 26307 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fcaaaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 1 -- 172.21.15.53:0/309014219 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8032670 con 0x7f7fc40b11f0 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fcaaaf640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fcaaaf640 4 set_mon_vals no callback set 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.814+0000 7f7fc37fe640 10 monclient: dump: 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 10 monclient: shutdown 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/309014219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40b11f0 msgr2=0x7f7fc413de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 --2- 172.21.15.53:0/309014219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40b11f0 0x7f7fc413de80 secure :-1 s=READY pgs=5495 cs=0 l=1 rev1=1 crypto rx=0x7f7fc406f590 tx=0x7f7fb80090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/309014219 shutdown_connections 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 --2- 172.21.15.53:0/309014219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40b11f0 0x7f7fc413de80 unknown :-1 s=CLOSED pgs=5495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 --2- 172.21.15.53:0/309014219 >> v2:172.21.15.104:3300/0 conn(0x7f7fc40b0e20 0x7f7fc40b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/309014219 >> 172.21.15.53:0/309014219 conn(0x7f7fc4062d90 msgr2=0x7f7fc4063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/309014219 shutdown_connections 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/309014219 wait complete. 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.815+0000 7f7fcaaaf640 10 monclient: monmap: 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 librados: starting msgr at 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 librados: starting objecter 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 Processor -- start 2024-09-15T02:58:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 -- start start 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 librados: setting wanted keys 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 librados: calling monclient init 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 10 monclient: init 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40a5900 0x7f7fc411c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc40a5900 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fc40b0e20 0x7f7fc411d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc40b0e20 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7fc40b11f0 0x7f7fc414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc40b11f0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.816+0000 7f7fcaaaf640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40adf80 con 0x7f7fc40a5900 2024-09-15T02:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc414d750 con 0x7f7fc40b0e20 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc411d690 con 0x7f7fc40b11f0 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient(hunting): authenticate will time out at 1640.387030s 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40a5900 0x7f7fc411c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40a5900 0x7f7fc411c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52498/0 (socket says 172.21.15.53:52498) 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 -- 172.21.15.53:0/4244951602 learned_addr learned my addr 172.21.15.53:0/4244951602 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): get_auth_request con 0x7f7fc40a5900 auth_method 0 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 1 --2- 172.21.15.53:0/4244951602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7fc40b11f0 0x7f7fc414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 10 monclient(hunting): get_auth_request con 0x7f7fc40b11f0 auth_method 0 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc3fff640 1 --2- 172.21.15.53:0/4244951602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fc40b0e20 0x7f7fc411d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc904c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc3fff640 10 monclient(hunting): get_auth_request con 0x7f7fc40b0e20 auth_method 0 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient(hunting): handle_auth_done global_id 26313 payload 824 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 -- 172.21.15.53:0/4244951602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7fc40b11f0 msgr2=0x7f7fc414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 --2- 172.21.15.53:0/4244951602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7fc40b11f0 0x7f7fc414dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 -- 172.21.15.53:0/4244951602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fc40b0e20 msgr2=0x7f7fc411d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 --2- 172.21.15.53:0/4244951602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fc40b0e20 0x7f7fc411d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 monclient: found mon.noname-a 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 -- 172.21.15.53:0/4244951602 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb8009390 con 0x7f7fc40a5900 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc3fff640 1 --2- 172.21.15.53:0/4244951602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fc40b0e20 0x7f7fc411d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:58:23.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc884b640 1 --2- 172.21.15.53:0/4244951602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40a5900 0x7f7fc411c3c0 secure :-1 s=READY pgs=5496 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c880 tx=0x7f7fb4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 client.?.objecter ms_handle_connect 0x7f7fc40a5900 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400ce10 con 0x7f7fc40a5900 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: dump: 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4007830 con 0x7f7fc40a5900 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 5 monclient: authenticate success, global_id 26313 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient: _renew_subs 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc411d8b0 con 0x7f7fc40a5900 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4007bf0 con 0x7f7fc40a5900 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc0ff9640 10 monclient: dump: 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 client.26313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient: _renew_subs 2024-09-15T02:58:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fc27fc640 4 set_mon_vals no callback set 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc40ac140 con 0x7f7fc40a5900 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.817+0000 7f7fcaaaf640 1 librados: init done 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.818+0000 7f7fcaaaf640 10 client.26313.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.818+0000 7f7fcaaaf640 15 client.26313.objecter _session_command_op_assign -1 1 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.818+0000 7f7fcaaaf640 20 client.26313.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.818+0000 7f7fcaaaf640 10 client.26313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.818+0000 7f7fcaaaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.818+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7fc40646e0 con 0x7f7fc40a5900 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f7fb4007d50 con 0x7f7fc40a5900 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6816/32804 conn(0x7f7fb006e830 0x7f7fb0070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f7fb408b860 con 0x7f7fc40a5900 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 10 client.26313.objecter ms_dispatch 0x7f7fc4000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:23.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 3 client.26313.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 3 client.26313.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 10 client.26313.objecter checking command 1 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6800/33346 conn(0x7f7fb0075dc0 0x7f7fb0078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.819+0000 7f7fc0ff9640 20 client.26313.objecter _get_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 20 client.26313.objecter put_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 15 client.26313.objecter _session_command_op_remove -1 1 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 20 client.26313.objecter _get_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 20 client.26313.objecter get_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 15 client.26313.objecter _session_command_op_assign 5 1 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 20 client.26313.objecter put_session s=0x7f7fb0075710 osd=5 4 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 10 client.26313.objecter _send_command 1 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 --> v2:172.21.15.104:6800/33346 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fb00788e0 con 0x7f7fb0075dc0 2024-09-15T02:58:23.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 20 client.26313.objecter dump_active .. 0 homeless 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fb4090240 con 0x7f7fc40a5900 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc3fff640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6816/32804 conn(0x7f7fb006e830 0x7f7fb0070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc3fff640 10 monclient: get_auth_request con 0x7f7fb006e830 auth_method 0 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc904c640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6800/33346 conn(0x7f7fb0075dc0 0x7f7fb0078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc904c640 10 monclient: get_auth_request con 0x7f7fb0075dc0 auth_method 0 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 10 client.26313.objecter ms_handle_connect 0x7f7fb006e830 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc3fff640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6816/32804 conn(0x7f7fb006e830 0x7f7fb0070ce0 secure :-1 s=READY pgs=3701 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8034000 tx=0x7f7fb8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc0ff9640 10 client.26313.objecter ms_handle_connect 0x7f7fb0075dc0 2024-09-15T02:58:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.820+0000 7f7fc904c640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6800/33346 conn(0x7f7fb0075dc0 0x7f7fb0078250 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.821+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== osd.5 v2:172.21.15.104:6800/33346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7fb00788e0 con 0x7f7fb0075dc0 2024-09-15T02:58:23.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.821+0000 7f7fc0ff9640 10 client.26313.objecter ms_dispatch 0x7f7fc4000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:23.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.821+0000 7f7fc0ff9640 10 client.26313.objecter _finish_command 1 = system:0 2024-09-15T02:58:23.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.821+0000 7f7fc0ff9640 20 client.26313.objecter put_session s=0x7f7fb0075710 osd=5 4 2024-09-15T02:58:23.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.821+0000 7f7fc0ff9640 15 client.26313.objecter _session_command_op_remove 5 1 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 10 client.26313.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 15 client.26313.objecter _session_command_op_assign -1 2 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 20 client.26313.objecter _get_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 20 client.26313.objecter put_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 20 client.26313.objecter _get_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 15 client.26313.objecter _session_command_op_remove -1 2 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 20 client.26313.objecter get_session s=0x7f7fb0075710 osd=5 3 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 15 client.26313.objecter _session_command_op_assign 5 2 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 20 client.26313.objecter put_session s=0x7f7fb0075710 osd=5 4 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 10 client.26313.objecter _send_command 2 2024-09-15T02:58:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.832+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 --> v2:172.21.15.104:6800/33346 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7fc40b1d50 con 0x7f7fb0075dc0 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fc0ff9640 1 -- 172.21.15.53:0/4244951602 <== osd.5 v2:172.21.15.104:6800/33346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7fc40b1d50 con 0x7f7fb0075dc0 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fc0ff9640 10 client.26313.objecter ms_dispatch 0x7f7fc4000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fc0ff9640 10 client.26313.objecter _finish_command 2 = system:0 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fc0ff9640 20 client.26313.objecter put_session s=0x7f7fb0075710 osd=5 4 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fc0ff9640 15 client.26313.objecter _session_command_op_remove 5 2 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:23.842 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 10 librados: watch_flush enter 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 10 librados: watch_flush exit 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 10 client.26313.objecter close_session for osd.5 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6800/33346 conn(0x7f7fb0075dc0 msgr2=0x7f7fb0078250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6800/33346 conn(0x7f7fb0075dc0 0x7f7fb0078250 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 20 client.26313.objecter put_session s=0x7f7fb0075710 osd=5 1 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 20 client.26313.objecter shutdown clearing up homeless session... 2024-09-15T02:58:23.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 10 client.26313.objecter successfully canceled tick 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6816/32804 conn(0x7f7fb006e830 msgr2=0x7f7fb0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 --2- 172.21.15.53:0/4244951602 >> v2:172.21.15.104:6816/32804 conn(0x7f7fb006e830 0x7f7fb0070ce0 secure :-1 s=READY pgs=3701 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8034000 tx=0x7f7fb8032000 comp rx=0 tx=0).stop 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 10 monclient: shutdown 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40a5900 msgr2=0x7f7fc411c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 --2- 172.21.15.53:0/4244951602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc40a5900 0x7f7fc411c3c0 secure :-1 s=READY pgs=5496 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c880 tx=0x7f7fb4008040 comp rx=0 tx=0).stop 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fc904c640 1 -- 172.21.15.53:0/4244951602 reap_dead start 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 shutdown_connections 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 >> 172.21.15.53:0/4244951602 conn(0x7f7fc4062d90 msgr2=0x7f7fc40afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 shutdown_connections 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.833+0000 7f7fcaaaf640 1 -- 172.21.15.53:0/4244951602 wait complete. 2024-09-15T02:58:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.834+0000 7f7fcaaaf640 1 librados: shutdown 2024-09-15T02:58:23.846 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T02:58:23.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.942+0000 7f932626f640 10 monclient: get_monmap_and_config 2024-09-15T02:58:23.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.942+0000 7f932626f640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.942+0000 7f932626f640 10 monclient: monmap: 2024-09-15T02:58:23.943 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.943 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:23.943401+0000 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:23.943401+0000 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:23.944 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.943+0000 7f932626f640 1 Processor -- start 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.943+0000 7f932626f640 1 -- start start 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.943+0000 7f932626f640 10 monclient: init 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f93200b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient(hunting): picked mon.noname-b con 0x7f9320140f70 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9320141340 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93200b0a80 0x7f93200b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient(hunting): picked mon.noname-a con 0x7f93200b0a80 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932013ddd0 con 0x7f9320141340 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932013e970 con 0x7f93200b0a80 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93200a6c00 con 0x7f9320140f70 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932626f640 10 monclient(hunting): authenticate will time out at 1640.514236s 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93200b0a80 0x7f93200b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f9317fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9320141340 0x7f932013e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59266/0 (socket says 172.21.15.53:59266) 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f9317fff640 1 -- 172.21.15.53:0/363207233 learned_addr learned my addr 172.21.15.53:0/363207233 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): get_auth_request con 0x7f93200b0a80 auth_method 0 2024-09-15T02:58:23.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 1 --2- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f93200b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 10 monclient(hunting): get_auth_request con 0x7f9320140f70 auth_method 0 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient(hunting): handle_auth_done global_id 26316 payload 274 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 1 -- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 msgr2=0x7f93200b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 1 --2- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f93200b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 1 -- 172.21.15.53:0/363207233 >> v1:172.21.15.53:6789/0 conn(0x7f9320141340 legacy=0x7f932013e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 1 monclient: found mon.noname-a 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931f7fe640 1 -- 172.21.15.53:0/363207233 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757937451 0 0) 0x7f932013ddd0 con 0x7f9320141340 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f932480c640 1 -- 172.21.15.53:0/363207233 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93200a5b30 con 0x7f93200b0a80 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931ffff640 1 --2- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f93200b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.944+0000 7f931f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f932480c640 1 --2- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f93200b0f00 secure :-1 s=READY pgs=5498 cs=0 l=1 rev1=1 crypto rx=0x7f9318009e60 tx=0x7f9318009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d850ad95d281f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 1 -- 172.21.15.53:0/363207233 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318007550 con 0x7f93200b0a80 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: dump: 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f932626f640 5 monclient: authenticate success, global_id 26316 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f932626f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 1 -- 172.21.15.53:0/363207233 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931802d050 con 0x7f93200b0a80 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 1 -- 172.21.15.53:0/363207233 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93180078e0 con 0x7f93200b0a80 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f932626f640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f932626f640 4 set_mon_vals no callback set 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.945+0000 7f931f7fe640 10 monclient: dump: 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.957 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 10 monclient: shutdown 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 -- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 msgr2=0x7f93200b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 --2- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f93200b0f00 secure :-1 s=READY pgs=5498 cs=0 l=1 rev1=1 crypto rx=0x7f9318009e60 tx=0x7f9318009780 comp rx=0 tx=0).stop 2024-09-15T02:58:23.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 -- 172.21.15.53:0/363207233 shutdown_connections 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 --2- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f93200b0f00 unknown :-1 s=CLOSED pgs=5498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 --2- 172.21.15.53:0/363207233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f93200b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 -- 172.21.15.53:0/363207233 >> 172.21.15.53:0/363207233 conn(0x7f9320062d90 msgr2=0x7f9320063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 -- 172.21.15.53:0/363207233 shutdown_connections 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 1 -- 172.21.15.53:0/363207233 wait complete. 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 10 monclient: build_initial_monmap 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.946+0000 7f932626f640 10 monclient: monmap: 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.959 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 librados: starting msgr at 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 librados: starting objecter 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 Processor -- start 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 -- start start 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 librados: setting wanted keys 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 librados: calling monclient init 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient: init 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f932013cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): picked mon.noname-a con 0x7f93200b0a80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f932013a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): picked mon.noname-c con 0x7f9320140f70 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9320141340 0x7f9320150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): picked mon.noname-b con 0x7f9320141340 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93200a7970 con 0x7f93200b0a80 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932013c640 con 0x7f9320141340 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932013c360 con 0x7f9320140f70 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f932626f640 10 monclient(hunting): authenticate will time out at 1640.517808s 2024-09-15T02:58:23.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f931ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f932013cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f9317fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f932013a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f931ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f932013cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52518/0 (socket says 172.21.15.53:52518) 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f931ffff640 1 -- 172.21.15.53:0/3156139277 learned_addr learned my addr 172.21.15.53:0/3156139277 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f931ffff640 10 monclient(hunting): get_auth_request con 0x7f93200b0a80 auth_method 0 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f931ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f931ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f931ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f9317fff640 10 monclient(hunting): get_auth_request con 0x7f9320140f70 auth_method 0 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f9317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f9317fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.947+0000 7f9317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 10 monclient(hunting): handle_auth_done global_id 20387 payload 824 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932480c640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9320141340 0x7f9320150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 1 -- 172.21.15.53:0/3156139277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9320141340 msgr2=0x7f9320150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9320141340 0x7f9320150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 1 -- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 msgr2=0x7f932013cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f932013cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 10 monclient: _finish_hunting 0 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 1 monclient: found mon.noname-c 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 1 -- 172.21.15.53:0/3156139277 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9310009390 con 0x7f9320140f70 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931ffff640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f932013cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f9317fff640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f932013a700 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f930800c490 tx=0x7f9308007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 client.?.objecter ms_handle_connect 0x7f9320140f70 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 1 -- 172.21.15.53:0/3156139277 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9308005aa0 con 0x7f9320140f70 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: dump: 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.964 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: _finish_auth 0 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 5 monclient: authenticate success, global_id 20387 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 1 -- 172.21.15.53:0/3156139277 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9308005c00 con 0x7f9320140f70 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 10 monclient: _renew_subs 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9318009a20 con 0x7f9320140f70 2024-09-15T02:58:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 1 -- 172.21.15.53:0/3156139277 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93080070d0 con 0x7f9320140f70 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931cff9640 10 monclient: dump: 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f931effd640 4 set_mon_vals no callback set 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 10 client.20387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 10 monclient: _renew_subs 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9320150a80 con 0x7f9320140f70 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.948+0000 7f932626f640 1 librados: init done 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.949+0000 7f932626f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.949+0000 7f932626f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.949+0000 7f932626f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.949+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932013c7f0 con 0x7f9320140f70 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931cff9640 1 -- 172.21.15.53:0/3156139277 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f9308007500 con 0x7f9320140f70 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931cff9640 1 --2- 172.21.15.53:0/3156139277 >> v2:172.21.15.104:6816/32804 conn(0x7f930006e800 0x7f9300070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931cff9640 1 -- 172.21.15.53:0/3156139277 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f930808ae80 con 0x7f9320140f70 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931cff9640 10 client.20387.objecter ms_dispatch 0x7f9320000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931cff9640 3 client.20387.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931cff9640 3 client.20387.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931cff9640 20 client.20387.objecter dump_active .. 0 homeless 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931ffff640 1 --2- 172.21.15.53:0/3156139277 >> v2:172.21.15.104:6816/32804 conn(0x7f930006e800 0x7f9300070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.950+0000 7f931ffff640 10 monclient: get_auth_request con 0x7f930006e800 auth_method 0 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.951+0000 7f931ffff640 1 --2- 172.21.15.53:0/3156139277 >> v2:172.21.15.104:6816/32804 conn(0x7f930006e800 0x7f9300070cb0 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7f9310011640 tx=0x7f93100090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.951+0000 7f931cff9640 10 client.20387.objecter ms_handle_connect 0x7f930006e800 2024-09-15T02:58:23.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.954+0000 7f931cff9640 1 -- 172.21.15.53:0/3156139277 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f932013c7f0 con 0x7f9320140f70 2024-09-15T02:58:23.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.954+0000 7f931cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:23.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:23.954+0000 7f931cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:24.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.118+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f932013c7f0 con 0x7f930006e800 2024-09-15T02:58:24.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.119+0000 7f931cff9640 1 -- 172.21.15.53:0/3156139277 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f932013c7f0 con 0x7f930006e800 2024-09-15T02:58:24.121 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T02:58:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 10 librados: watch_flush enter 2024-09-15T02:58:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 10 librados: watch_flush exit 2024-09-15T02:58:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 20 client.20387.objecter shutdown clearing up homeless session... 2024-09-15T02:58:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 10 client.20387.objecter successfully canceled tick 2024-09-15T02:58:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 >> v2:172.21.15.104:6816/32804 conn(0x7f930006e800 msgr2=0x7f9300070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 --2- 172.21.15.53:0/3156139277 >> v2:172.21.15.104:6816/32804 conn(0x7f930006e800 0x7f9300070cb0 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7f9310011640 tx=0x7f93100090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 10 monclient: shutdown 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 msgr2=0x7f932013a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f932013a700 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f930800c490 tx=0x7f9308007ec0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 shutdown_connections 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 --2- 172.21.15.53:0/3156139277 >> v2:172.21.15.104:6816/32804 conn(0x7f930006e800 0x7f9300070cb0 unknown :-1 s=CLOSED pgs=3702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9320141340 0x7f9320150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9320140f70 0x7f932013a700 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 --2- 172.21.15.53:0/3156139277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93200b0a80 0x7f932013cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 >> 172.21.15.53:0/3156139277 conn(0x7f9320062d90 msgr2=0x7f932013e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 shutdown_connections 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 -- 172.21.15.53:0/3156139277 wait complete. 2024-09-15T02:58:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.121+0000 7f932626f640 1 librados: shutdown 2024-09-15T02:58:24.133 INFO:tasks.ceph:Scrubbing osd.6 2024-09-15T02:58:24.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:24.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.231+0000 7f2ac8d5d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.231+0000 7f2ac8d5d640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.231+0000 7f2ac8d5d640 10 monclient: monmap: 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:24.232744+0000 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:24.232744+0000 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:24.233 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:24.234 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:24.234 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.232+0000 7f2ac8d5d640 1 Processor -- start 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.232+0000 7f2ac8d5d640 1 -- start start 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.232+0000 7f2ac8d5d640 10 monclient: init 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac40b0fa0 0x7f2ac413a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac40b0fa0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2ac40a5920 0x7f2ac413aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac40a5920 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ac413dd30 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:24.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40aed10 con 0x7f2ac413dd30 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac40a6b20 con 0x7f2ac40a5920 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac413e8c0 con 0x7f2ac40b0fa0 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac8d5d640 10 monclient(hunting): authenticate will time out at 1640.803271s 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac40b0fa0 0x7f2ac413a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac2d9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2ac413dd30 0x7f2ac413e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59280/0 (socket says 172.21.15.53:59280) 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac2d9d640 1 -- 172.21.15.53:0/2337725877 learned_addr learned my addr 172.21.15.53:0/2337725877 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 1 --2- 172.21.15.53:0/2337725877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac40b0fa0 0x7f2ac413a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56416/0 (socket says 172.21.15.53:56416) 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): get_auth_request con 0x7f2ac40b0fa0 auth_method 0 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ab9d9b640 1 --2- 172.21.15.53:0/2337725877 >> v2:172.21.15.104:3300/0 conn(0x7f2ac40a5920 0x7f2ac413aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ab9d9b640 10 monclient(hunting): get_auth_request con 0x7f2ac40a5920 auth_method 0 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ab9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ab9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ab9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient(hunting): handle_auth_done global_id 20390 payload 274 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 1 -- 172.21.15.53:0/2337725877 >> v2:172.21.15.104:3300/0 conn(0x7f2ac40a5920 msgr2=0x7f2ac413aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 1 --2- 172.21.15.53:0/2337725877 >> v2:172.21.15.104:3300/0 conn(0x7f2ac40a5920 0x7f2ac413aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 1 -- 172.21.15.53:0/2337725877 >> v1:172.21.15.53:6789/0 conn(0x7f2ac413dd30 legacy=0x7f2ac413e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac1d9b640 1 -- 172.21.15.53:0/2337725877 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172200725 0 0) 0x7f2ac40aed10 con 0x7f2ac413dd30 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 1 monclient: found mon.noname-b 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac259c640 1 -- 172.21.15.53:0/2337725877 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40b1370 con 0x7f2ac40b0fa0 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ab9d9b640 1 --2- 172.21.15.53:0/2337725877 >> v2:172.21.15.104:3300/0 conn(0x7f2ac40a5920 0x7f2ac413aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.233+0000 7f2ac1d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac259c640 1 --2- 172.21.15.53:0/2337725877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac40b0fa0 0x7f2ac413a9b0 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009850 tx=0x7f2ab40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34bd0ae67b39d386 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 1 -- 172.21.15.53:0/2337725877 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab402ad00 con 0x7f2ac40b0fa0 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: dump: 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 1 -- 172.21.15.53:0/2337725877 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab402ae60 con 0x7f2ac40b0fa0 2024-09-15T02:58:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac8d5d640 5 monclient: authenticate success, global_id 20390 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac8d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac8d5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 1 -- 172.21.15.53:0/2337725877 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4032670 con 0x7f2ac40b0fa0 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac8d5d640 4 set_mon_vals no callback set 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.234+0000 7f2ac1d9b640 10 monclient: dump: 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 10 monclient: shutdown 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/2337725877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac40b0fa0 msgr2=0x7f2ac413a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 --2- 172.21.15.53:0/2337725877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac40b0fa0 0x7f2ac413a9b0 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009850 tx=0x7f2ab40090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/2337725877 shutdown_connections 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 --2- 172.21.15.53:0/2337725877 >> v2:172.21.15.104:3300/0 conn(0x7f2ac40a5920 0x7f2ac413aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 --2- 172.21.15.53:0/2337725877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac40b0fa0 0x7f2ac413a9b0 unknown :-1 s=CLOSED pgs=3140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/2337725877 >> 172.21.15.53:0/2337725877 conn(0x7f2ac4062d90 msgr2=0x7f2ac4063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/2337725877 shutdown_connections 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/2337725877 wait complete. 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.235+0000 7f2ac8d5d640 10 monclient: monmap: 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:24.249 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 librados: starting msgr at 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 librados: starting objecter 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 Processor -- start 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 -- start start 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 librados: setting wanted keys 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 librados: calling monclient init 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient: init 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ac40a5920 0x7f2ac4141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac40a5920 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ac40b0fa0 0x7f2ac40a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac40b0fa0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac413dd30 0x7f2ac41370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac413dd30 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac413f180 con 0x7f2ac40a5920 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac40a8ae0 con 0x7f2ac40b0fa0 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac40a8d50 con 0x7f2ac413dd30 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac8d5d640 10 monclient(hunting): authenticate will time out at 1640.806821s 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac2d9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac413dd30 0x7f2ac41370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ac40a5920 0x7f2ac4141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ac40a5920 0x7f2ac4141920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52524/0 (socket says 172.21.15.53:52524) 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac2d9d640 10 monclient(hunting): get_auth_request con 0x7f2ac413dd30 auth_method 0 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac259c640 1 -- 172.21.15.53:0/343966299 learned_addr learned my addr 172.21.15.53:0/343966299 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac259c640 10 monclient(hunting): get_auth_request con 0x7f2ac40a5920 auth_method 0 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.236+0000 7f2ac259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ab9d9b640 1 --2- 172.21.15.53:0/343966299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ac40b0fa0 0x7f2ac40a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 10 monclient(hunting): handle_auth_done global_id 20393 payload 824 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 1 -- 172.21.15.53:0/343966299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ac40b0fa0 msgr2=0x7f2ac40a76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 1 --2- 172.21.15.53:0/343966299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ac40b0fa0 0x7f2ac40a76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 1 -- 172.21.15.53:0/343966299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ac40a5920 msgr2=0x7f2ac4141920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 1 --2- 172.21.15.53:0/343966299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ac40a5920 0x7f2ac4141920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 1 monclient: found mon.noname-c 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 1 -- 172.21.15.53:0/343966299 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4009390 con 0x7f2ac413dd30 2024-09-15T02:58:24.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ab9d9b640 10 monclient: get_auth_request con 0x7f2ac40b0fa0 auth_method 0 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ab9d9b640 1 --2- 172.21.15.53:0/343966299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ac40b0fa0 0x7f2ac40a76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac259c640 1 --2- 172.21.15.53:0/343966299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ac40a5920 0x7f2ac4141920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac2d9d640 1 --2- 172.21.15.53:0/343966299 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac413dd30 0x7f2ac41370c0 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00cbc0 tx=0x7f2abc0018f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 client.?.objecter ms_handle_connect 0x7f2ac413dd30 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00a5e0 con 0x7f2ac413dd30 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: dump: 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.254 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc00a760 con 0x7f2ac413dd30 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 5 monclient: authenticate success, global_id 20393 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 10 monclient: _renew_subs 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac40a7bf0 con 0x7f2ac413dd30 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00ab60 con 0x7f2ac413dd30 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2abaffd640 10 monclient: dump: 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 10 client.20393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 10 monclient: _renew_subs 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac159a640 4 set_mon_vals no callback set 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac413f610 con 0x7f2ac413dd30 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.237+0000 7f2ac8d5d640 1 librados: init done 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.238+0000 7f2ac8d5d640 10 client.20393.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.238+0000 7f2ac8d5d640 15 client.20393.objecter _session_command_op_assign -1 1 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.238+0000 7f2ac8d5d640 20 client.20393.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.238+0000 7f2ac8d5d640 10 client.20393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.238+0000 7f2ac8d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.238+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2ac40646e0 con 0x7f2ac413dd30 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f2abc0070f0 con 0x7f2ac413dd30 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6816/32804 conn(0x7f2aa406e800 0x7f2aa4070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f2abc08b9b0 con 0x7f2ac413dd30 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 10 client.20393.objecter ms_dispatch 0x7f2ac4000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 3 client.20393.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 3 client.20393.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 10 client.20393.objecter checking command 1 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6808/33357 conn(0x7f2aa4075d90 0x7f2aa4078220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 20 client.20393.objecter _get_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 20 client.20393.objecter put_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 15 client.20393.objecter _session_command_op_remove -1 1 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 20 client.20393.objecter _get_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 20 client.20393.objecter get_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 15 client.20393.objecter _session_command_op_assign 6 1 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 20 client.20393.objecter put_session s=0x7f2aa40756e0 osd=6 4 2024-09-15T02:58:24.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 10 client.20393.objecter _send_command 1 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 --> v2:172.21.15.104:6808/33357 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2aa40788b0 con 0x7f2aa4075d90 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 20 client.20393.objecter dump_active .. 0 homeless 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2abc090220 con 0x7f2ac413dd30 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2abaffd640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2ac259c640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6816/32804 conn(0x7f2aa406e800 0x7f2aa4070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2ac259c640 10 monclient: get_auth_request con 0x7f2aa406e800 auth_method 0 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2ab9d9b640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6808/33357 conn(0x7f2aa4075d90 0x7f2aa4078220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.239+0000 7f2ab9d9b640 10 monclient: get_auth_request con 0x7f2aa4075d90 auth_method 0 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.240+0000 7f2abaffd640 10 client.20393.objecter ms_handle_connect 0x7f2aa406e800 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.240+0000 7f2ac259c640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6816/32804 conn(0x7f2aa406e800 0x7f2aa4070cb0 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40a89a0 tx=0x7f2ab4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.240+0000 7f2abaffd640 10 client.20393.objecter ms_handle_connect 0x7f2aa4075d90 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.240+0000 7f2ab9d9b640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6808/33357 conn(0x7f2aa4075d90 0x7f2aa4078220 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.241+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== osd.6 v2:172.21.15.104:6808/33357 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2aa40788b0 con 0x7f2aa4075d90 2024-09-15T02:58:24.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.241+0000 7f2abaffd640 10 client.20393.objecter ms_dispatch 0x7f2ac4000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:24.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.241+0000 7f2abaffd640 10 client.20393.objecter _finish_command 1 = system:0 2024-09-15T02:58:24.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.241+0000 7f2abaffd640 20 client.20393.objecter put_session s=0x7f2aa40756e0 osd=6 4 2024-09-15T02:58:24.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.241+0000 7f2abaffd640 15 client.20393.objecter _session_command_op_remove 6 1 2024-09-15T02:58:24.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 10 client.20393.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 15 client.20393.objecter _session_command_op_assign -1 2 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 client.20393.objecter _get_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 client.20393.objecter put_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 client.20393.objecter _get_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 15 client.20393.objecter _session_command_op_remove -1 2 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 client.20393.objecter get_session s=0x7f2aa40756e0 osd=6 3 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 15 client.20393.objecter _session_command_op_assign 6 2 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 client.20393.objecter put_session s=0x7f2aa40756e0 osd=6 4 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 10 client.20393.objecter _send_command 2 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 --> v2:172.21.15.104:6808/33357 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2ac40a7e40 con 0x7f2aa4075d90 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2abaffd640 1 -- 172.21.15.53:0/343966299 <== osd.6 v2:172.21.15.104:6808/33357 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2ac40a7e40 con 0x7f2aa4075d90 2024-09-15T02:58:24.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2abaffd640 10 client.20393.objecter ms_dispatch 0x7f2ac4000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:24.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2abaffd640 10 client.20393.objecter _finish_command 2 = system:0 2024-09-15T02:58:24.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2abaffd640 20 client.20393.objecter put_session s=0x7f2aa40756e0 osd=6 4 2024-09-15T02:58:24.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2abaffd640 15 client.20393.objecter _session_command_op_remove 6 2 2024-09-15T02:58:24.262 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:24.262 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:24.262 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:24.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 10 librados: watch_flush enter 2024-09-15T02:58:24.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 10 librados: watch_flush exit 2024-09-15T02:58:24.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 10 client.20393.objecter close_session for osd.6 2024-09-15T02:58:24.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6808/33357 conn(0x7f2aa4075d90 msgr2=0x7f2aa4078220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6808/33357 conn(0x7f2aa4075d90 0x7f2aa4078220 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 client.20393.objecter put_session s=0x7f2aa40756e0 osd=6 1 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 client.20393.objecter shutdown clearing up homeless session... 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 10 client.20393.objecter successfully canceled tick 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6816/32804 conn(0x7f2aa406e800 msgr2=0x7f2aa4070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 1 --2- 172.21.15.53:0/343966299 >> v2:172.21.15.104:6816/32804 conn(0x7f2aa406e800 0x7f2aa4070cb0 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40a89a0 tx=0x7f2ab4032000 comp rx=0 tx=0).stop 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 10 monclient: shutdown 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac413dd30 msgr2=0x7f2ac41370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac8d5d640 1 --2- 172.21.15.53:0/343966299 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ac413dd30 0x7f2ac41370c0 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00cbc0 tx=0x7f2abc0018f0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.253+0000 7f2ac2d9d640 1 -- 172.21.15.53:0/343966299 reap_dead start 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.254+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 shutdown_connections 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.254+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 >> 172.21.15.53:0/343966299 conn(0x7f2ac4062d90 msgr2=0x7f2ac40aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.254+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 shutdown_connections 2024-09-15T02:58:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.254+0000 7f2ac8d5d640 1 -- 172.21.15.53:0/343966299 wait complete. 2024-09-15T02:58:24.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.254+0000 7f2ac8d5d640 1 librados: shutdown 2024-09-15T02:58:24.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.362+0000 7f9a2feef640 10 monclient: get_monmap_and_config 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.362+0000 7f9a2feef640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.362+0000 7f9a2feef640 10 monclient: monmap: 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:24.363212+0000 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:24.363212+0000 2024-09-15T02:58:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.364 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.364 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:24.364 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:24.364 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:24.364 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:24.364 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:24.364 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 1 Processor -- start 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 1 -- start start 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient: init 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient(hunting): picked mon.noname-b con 0x7f9a28140f70 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:24.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a28141340 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a280b0a80 0x7f9a280b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient(hunting): picked mon.noname-c con 0x7f9a280b0a80 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2813ddd0 con 0x7f9a28141340 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2813e970 con 0x7f9a280b0a80 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280a6c00 con 0x7f9a28140f70 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.363+0000 7f9a2feef640 10 monclient(hunting): authenticate will time out at 1640.933905s 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2d48a640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9a28141340 0x7f9a2813e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59282/0 (socket says 172.21.15.53:59282) 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2d48a640 1 -- 172.21.15.53:0/2659553830 learned_addr learned my addr 172.21.15.53:0/2659553830 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:24.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): get_auth_request con 0x7f9a28140f70 auth_method 0 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2e48c640 1 --2- 172.21.15.53:0/2659553830 >> v2:172.21.15.104:3300/0 conn(0x7f9a280b0a80 0x7f9a280b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2e48c640 10 monclient(hunting): get_auth_request con 0x7f9a280b0a80 auth_method 0 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2e48c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2e48c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2e48c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient(hunting): handle_auth_done global_id 20396 payload 274 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2cc89640 1 -- 172.21.15.53:0/2659553830 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988258520 0 0) 0x7f9a2813ddd0 con 0x7f9a28141340 2024-09-15T02:58:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 1 -- 172.21.15.53:0/2659553830 >> v2:172.21.15.104:3300/0 conn(0x7f9a280b0a80 msgr2=0x7f9a280b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 1 --2- 172.21.15.53:0/2659553830 >> v2:172.21.15.104:3300/0 conn(0x7f9a280b0a80 0x7f9a280b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 1 -- 172.21.15.53:0/2659553830 >> v1:172.21.15.53:6789/0 conn(0x7f9a28141340 legacy=0x7f9a2813e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 1 monclient: found mon.noname-b 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2dc8b640 1 -- 172.21.15.53:0/2659553830 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a280a5b30 con 0x7f9a28140f70 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.364+0000 7f9a2cc89640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2dc8b640 1 --2- 172.21.15.53:0/2659553830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280b00f0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009850 tx=0x7f9a1c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac760c4bc26cee6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 1 -- 172.21.15.53:0/2659553830 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c02ad00 con 0x7f9a28140f70 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: dump: 2024-09-15T02:58:24.375 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 1 -- 172.21.15.53:0/2659553830 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1c02ae60 con 0x7f9a28140f70 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 5 monclient: authenticate success, global_id 20396 2024-09-15T02:58:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 1 -- 172.21.15.53:0/2659553830 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c032670 con 0x7f9a28140f70 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 4 set_mon_vals no callback set 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2cc89640 10 monclient: dump: 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.377 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 10 monclient: shutdown 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 -- 172.21.15.53:0/2659553830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 msgr2=0x7f9a280b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 --2- 172.21.15.53:0/2659553830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280b00f0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009850 tx=0x7f9a1c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 -- 172.21.15.53:0/2659553830 shutdown_connections 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 --2- 172.21.15.53:0/2659553830 >> v2:172.21.15.104:3300/0 conn(0x7f9a280b0a80 0x7f9a280b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 --2- 172.21.15.53:0/2659553830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280b00f0 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 -- 172.21.15.53:0/2659553830 >> 172.21.15.53:0/2659553830 conn(0x7f9a28062d90 msgr2=0x7f9a28063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 -- 172.21.15.53:0/2659553830 shutdown_connections 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.365+0000 7f9a2feef640 1 -- 172.21.15.53:0/2659553830 wait complete. 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 10 monclient: monmap: 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:24.379 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 1 librados: starting msgr at 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 1 librados: starting objecter 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 1 Processor -- start 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 1 -- start start 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 1 librados: setting wanted keys 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 1 librados: calling monclient init 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.366+0000 7f9a2feef640 10 monclient: init 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a280b0a80 0x7f9a2813d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): picked mon.noname-a con 0x7f9a280b0a80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280a7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): picked mon.noname-c con 0x7f9a28140f70 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a28141340 0x7f9a280a7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): picked mon.noname-b con 0x7f9a28141340 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280aa1c0 con 0x7f9a280b0a80 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280ab810 con 0x7f9a28141340 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280a8570 con 0x7f9a28140f70 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2feef640 10 monclient(hunting): authenticate will time out at 1640.937415s 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a280b0a80 0x7f9a2813d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280a7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56448/0 (socket says 172.21.15.53:56448) 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 -- 172.21.15.53:0/168690637 learned_addr learned my addr 172.21.15.53:0/168690637 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): get_auth_request con 0x7f9a28140f70 auth_method 0 2024-09-15T02:58:24.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a280b0a80 0x7f9a2813d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52540/0 (socket says 172.21.15.53:52540) 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 10 monclient(hunting): get_auth_request con 0x7f9a280b0a80 auth_method 0 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a28141340 0x7f9a280a7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2dc8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 10 monclient(hunting): get_auth_request con 0x7f9a28141340 auth_method 0 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2e48c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient(hunting): handle_auth_done global_id 20399 payload 824 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 -- 172.21.15.53:0/168690637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a28141340 msgr2=0x7f9a280a7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a28141340 0x7f9a280a7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 -- 172.21.15.53:0/168690637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a280b0a80 msgr2=0x7f9a2813d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a280b0a80 0x7f9a2813d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 monclient: found mon.noname-c 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.367+0000 7f9a2d48a640 1 -- 172.21.15.53:0/168690637 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c009390 con 0x7f9a28140f70 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2d48a640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280a7320 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c510 tx=0x7f9a2400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 client.?.objecter ms_handle_connect 0x7f9a28140f70 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 1 -- 172.21.15.53:0/168690637 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24005aa0 con 0x7f9a28140f70 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: dump: 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.384 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 1 -- 172.21.15.53:0/168690637 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a24005c00 con 0x7f9a28140f70 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 5 monclient: authenticate success, global_id 20399 2024-09-15T02:58:24.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 10 monclient: _renew_subs 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a280a9140 con 0x7f9a28140f70 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 1 -- 172.21.15.53:0/168690637 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a240070d0 con 0x7f9a28140f70 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a19ffb640 10 monclient: dump: 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a1bfff640 4 set_mon_vals no callback set 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 10 client.20399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 10 monclient: _renew_subs 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a280a9e40 con 0x7f9a28140f70 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.368+0000 7f9a2feef640 1 librados: init done 2024-09-15T02:58:24.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.369+0000 7f9a2feef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.369+0000 7f9a2feef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.369+0000 7f9a2feef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.369+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a280aa330 con 0x7f9a28140f70 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.369+0000 7f9a19ffb640 1 -- 172.21.15.53:0/168690637 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f9a24007500 con 0x7f9a28140f70 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a19ffb640 1 --2- 172.21.15.53:0/168690637 >> v2:172.21.15.104:6816/32804 conn(0x7f9a0c06e800 0x7f9a0c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a19ffb640 1 -- 172.21.15.53:0/168690637 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 447..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f9a2408af90 con 0x7f9a28140f70 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a19ffb640 10 client.20399.objecter ms_dispatch 0x7f9a28000b90 osd_map(702..702 src has 447..702) v4 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a19ffb640 3 client.20399.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a19ffb640 3 client.20399.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a19ffb640 20 client.20399.objecter dump_active .. 0 homeless 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a2dc8b640 1 --2- 172.21.15.53:0/168690637 >> v2:172.21.15.104:6816/32804 conn(0x7f9a0c06e800 0x7f9a0c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.370+0000 7f9a2dc8b640 10 monclient: get_auth_request con 0x7f9a0c06e800 auth_method 0 2024-09-15T02:58:24.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.371+0000 7f9a2dc8b640 1 --2- 172.21.15.53:0/168690637 >> v2:172.21.15.104:6816/32804 conn(0x7f9a0c06e800 0x7f9a0c070cb0 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c034000 tx=0x7f9a1c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.371+0000 7f9a19ffb640 10 client.20399.objecter ms_handle_connect 0x7f9a0c06e800 2024-09-15T02:58:24.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.374+0000 7f9a19ffb640 1 -- 172.21.15.53:0/168690637 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9a24055230 con 0x7f9a28140f70 2024-09-15T02:58:24.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.374+0000 7f9a19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.374+0000 7f9a19ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.541+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a280646e0 con 0x7f9a0c06e800 2024-09-15T02:58:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.541+0000 7f9a19ffb640 1 -- 172.21.15.53:0/168690637 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9a280646e0 con 0x7f9a0c06e800 2024-09-15T02:58:24.542 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 6 to deep-scrub 2024-09-15T02:58:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.543+0000 7f9a2feef640 10 librados: watch_flush enter 2024-09-15T02:58:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.543+0000 7f9a2feef640 10 librados: watch_flush exit 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.543+0000 7f9a2feef640 20 client.20399.objecter shutdown clearing up homeless session... 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.543+0000 7f9a2feef640 10 client.20399.objecter successfully canceled tick 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 >> v2:172.21.15.104:6816/32804 conn(0x7f9a0c06e800 msgr2=0x7f9a0c070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 --2- 172.21.15.53:0/168690637 >> v2:172.21.15.104:6816/32804 conn(0x7f9a0c06e800 0x7f9a0c070cb0 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c034000 tx=0x7f9a1c032000 comp rx=0 tx=0).stop 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 10 monclient: shutdown 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 msgr2=0x7f9a280a7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280a7320 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c510 tx=0x7f9a2400c910 comp rx=0 tx=0).stop 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 shutdown_connections 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 --2- 172.21.15.53:0/168690637 >> v2:172.21.15.104:6816/32804 conn(0x7f9a0c06e800 0x7f9a0c070cb0 unknown :-1 s=CLOSED pgs=3704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a28141340 0x7f9a280a7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a28140f70 0x7f9a280a7320 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 --2- 172.21.15.53:0/168690637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a280b0a80 0x7f9a2813d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 >> 172.21.15.53:0/168690637 conn(0x7f9a28062d90 msgr2=0x7f9a280b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 shutdown_connections 2024-09-15T02:58:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 -- 172.21.15.53:0/168690637 wait complete. 2024-09-15T02:58:24.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.544+0000 7f9a2feef640 1 librados: shutdown 2024-09-15T02:58:24.557 INFO:tasks.ceph:Scrubbing osd.7 2024-09-15T02:58:24.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.658+0000 7f12ae2d4640 10 monclient: get_monmap_and_config 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.658+0000 7f12ae2d4640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.658+0000 7f12ae2d4640 10 monclient: monmap: 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:24.658938+0000 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:24.658938+0000 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:24.659 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:24.660 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.658+0000 7f12ae2d4640 1 Processor -- start 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.658+0000 7f12ae2d4640 1 -- start start 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.658+0000 7f12ae2d4640 10 monclient: init 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a8068010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient(hunting): picked mon.noname-b con 0x7f12a80b2c40 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12a8068c50 0x7f12a8140e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8068c50 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12a806df40 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a806ec40 con 0x7f12a806df40 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a8068550 con 0x7f12a8068c50 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8069cb0 con 0x7f12a80b2c40 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ae2d4640 10 monclient(hunting): authenticate will time out at 1641.229627s 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a8068010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ac871640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f12a806df40 0x7f12a80acd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59288/0 (socket says 172.21.15.53:59288) 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a8068010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56464/0 (socket says 172.21.15.53:56464) 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12ac871640 1 -- 172.21.15.53:0/1158431264 learned_addr learned my addr 172.21.15.53:0/1158431264 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 10 monclient(hunting): get_auth_request con 0x7f12a80b2c40 auth_method 0 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a77fe640 1 --2- 172.21.15.53:0/1158431264 >> v2:172.21.15.104:3300/0 conn(0x7f12a8068c50 0x7f12a8140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a77fe640 10 monclient(hunting): get_auth_request con 0x7f12a8068c50 auth_method 0 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.659+0000 7f12a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 10 monclient(hunting): handle_auth_done global_id 20402 payload 274 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a6ffd640 1 -- 172.21.15.53:0/1158431264 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730299012 0 0) 0x7f12a806ec40 con 0x7f12a806df40 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 1 -- 172.21.15.53:0/1158431264 >> v2:172.21.15.104:3300/0 conn(0x7f12a8068c50 msgr2=0x7f12a8140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 1 --2- 172.21.15.53:0/1158431264 >> v2:172.21.15.104:3300/0 conn(0x7f12a8068c50 0x7f12a8140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 1 -- 172.21.15.53:0/1158431264 >> v1:172.21.15.53:6789/0 conn(0x7f12a806df40 legacy=0x7f12a80acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 1 monclient: found mon.noname-b 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 1 -- 172.21.15.53:0/1158431264 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8141340 con 0x7f12a80b2c40 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a7fff640 1 --2- 172.21.15.53:0/1158431264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a8068010 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f1298009850 tx=0x7f12980090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0777a46a627697 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a6ffd640 1 -- 172.21.15.53:0/1158431264 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129802ac80 con 0x7f12a80b2c40 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.660+0000 7f12a6ffd640 10 monclient: dump: 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 5 monclient: authenticate success, global_id 20402 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 1 -- 172.21.15.53:0/1158431264 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129802ade0 con 0x7f12a80b2c40 2024-09-15T02:58:24.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 4 set_mon_vals no callback set 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 1 -- 172.21.15.53:0/1158431264 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298032670 con 0x7f12a80b2c40 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12a6ffd640 10 monclient: dump: 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.673 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 10 monclient: shutdown 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 -- 172.21.15.53:0/1158431264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 msgr2=0x7f12a8068010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 --2- 172.21.15.53:0/1158431264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a8068010 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f1298009850 tx=0x7f12980090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 -- 172.21.15.53:0/1158431264 shutdown_connections 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 --2- 172.21.15.53:0/1158431264 >> v2:172.21.15.104:3300/0 conn(0x7f12a8068c50 0x7f12a8140e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 --2- 172.21.15.53:0/1158431264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a8068010 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 -- 172.21.15.53:0/1158431264 >> 172.21.15.53:0/1158431264 conn(0x7f12a8063810 msgr2=0x7f12a80b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 -- 172.21.15.53:0/1158431264 shutdown_connections 2024-09-15T02:58:24.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.661+0000 7f12ae2d4640 1 -- 172.21.15.53:0/1158431264 wait complete. 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 10 monclient: monmap: 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 1 librados: starting msgr at 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 1 librados: starting objecter 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 1 Processor -- start 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 1 -- start start 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 1 librados: setting wanted keys 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 1 librados: calling monclient init 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.662+0000 7f12ae2d4640 10 monclient: init 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a8068c50 0x7f12a813f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): picked mon.noname-b con 0x7f12a8068c50 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a806df40 0x7f12a813e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): picked mon.noname-a con 0x7f12a806df40 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:24.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a813a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): picked mon.noname-c con 0x7f12a80b2c40 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a81408c0 con 0x7f12a806df40 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a813bff0 con 0x7f12a8068c50 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8140ac0 con 0x7f12a80b2c40 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ae2d4640 10 monclient(hunting): authenticate will time out at 1641.233245s 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a813a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a806df40 0x7f12a813e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a806df40 0x7f12a813e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52556/0 (socket says 172.21.15.53:52556) 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a813a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56466/0 (socket says 172.21.15.53:56466) 2024-09-15T02:58:24.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 1 -- 172.21.15.53:0/2451223235 learned_addr learned my addr 172.21.15.53:0/2451223235 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 10 monclient(hunting): get_auth_request con 0x7f12a806df40 auth_method 0 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): get_auth_request con 0x7f12a80b2c40 auth_method 0 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 1 --2- 172.21.15.53:0/2451223235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a8068c50 0x7f12a813f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 10 monclient(hunting): get_auth_request con 0x7f12a8068c50 auth_method 0 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient(hunting): handle_auth_done global_id 20405 payload 824 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 -- 172.21.15.53:0/2451223235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a8068c50 msgr2=0x7f12a813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 --2- 172.21.15.53:0/2451223235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a8068c50 0x7f12a813f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 -- 172.21.15.53:0/2451223235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a806df40 msgr2=0x7f12a813e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 --2- 172.21.15.53:0/2451223235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a806df40 0x7f12a813e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 monclient: found mon.noname-c 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 -- 172.21.15.53:0/2451223235 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298009390 con 0x7f12a80b2c40 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12a77fe640 1 --2- 172.21.15.53:0/2451223235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a806df40 0x7f12a813e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f12ac871640 1 --2- 172.21.15.53:0/2451223235 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a813a000 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f12a000cbe0 tx=0x7f12a00019a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f128ffff640 10 client.?.objecter ms_handle_connect 0x7f12a80b2c40 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f128ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.663+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a000ce00 con 0x7f12a80b2c40 2024-09-15T02:58:24.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: dump: 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a000a850 con 0x7f12a80b2c40 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a000ac90 con 0x7f12a80b2c40 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 5 monclient: authenticate success, global_id 20405 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 monclient: _renew_subs 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a813ee40 con 0x7f12a80b2c40 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f128ffff640 10 monclient: dump: 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.682 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12a67fc640 4 set_mon_vals no callback set 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 client.20405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 monclient: _renew_subs 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a813a7b0 con 0x7f12a80b2c40 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 1 librados: init done 2024-09-15T02:58:24.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 client.20405.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 15 client.20405.objecter _session_command_op_assign -1 1 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 20 client.20405.objecter _get_session osd=-1 returning homeless 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 client.20405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.664+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f12a80646e0 con 0x7f12a80b2c40 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.665+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f12a0007300 con 0x7f12a80b2c40 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.665+0000 7f128ffff640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6816/32804 conn(0x7f129406e800 0x7f1294070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 502..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f12a008ba60 con 0x7f12a80b2c40 2024-09-15T02:58:24.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 10 client.20405.objecter ms_dispatch 0x7f12a8000b90 osd_map(702..702 src has 502..702) v4 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 3 client.20405.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 3 client.20405.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 10 client.20405.objecter checking command 1 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6804/33347 conn(0x7f1294075d90 0x7f1294078220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 20 client.20405.objecter _get_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 20 client.20405.objecter put_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 15 client.20405.objecter _session_command_op_remove -1 1 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 20 client.20405.objecter _get_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 20 client.20405.objecter get_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 15 client.20405.objecter _session_command_op_assign 7 1 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 20 client.20405.objecter put_session s=0x7f12940756e0 osd=7 4 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 10 client.20405.objecter _send_command 1 2024-09-15T02:58:24.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 --> v2:172.21.15.104:6804/33347 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12940788b0 con 0x7f1294075d90 2024-09-15T02:58:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 20 client.20405.objecter dump_active .. 0 homeless 2024-09-15T02:58:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=702) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12a0090240 con 0x7f12a80b2c40 2024-09-15T02:58:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 10 monclient: handle_get_version_reply finishing 1 version 702 2024-09-15T02:58:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f12a7fff640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6816/32804 conn(0x7f129406e800 0x7f1294070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f12a7fff640 10 monclient: get_auth_request con 0x7f129406e800 auth_method 0 2024-09-15T02:58:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f12a77fe640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6804/33347 conn(0x7f1294075d90 0x7f1294078220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f12a77fe640 10 monclient: get_auth_request con 0x7f1294075d90 auth_method 0 2024-09-15T02:58:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 10 client.20405.objecter ms_handle_connect 0x7f1294075d90 2024-09-15T02:58:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f12a77fe640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6804/33347 conn(0x7f1294075d90 0x7f1294078220 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f128ffff640 10 client.20405.objecter ms_handle_connect 0x7f129406e800 2024-09-15T02:58:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.666+0000 7f12a7fff640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6816/32804 conn(0x7f129406e800 0x7f1294070cb0 secure :-1 s=READY pgs=3705 cs=0 l=1 rev1=1 crypto rx=0x7f1298009850 tx=0x7f1298032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.667+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== osd.7 v2:172.21.15.104:6804/33347 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f12940788b0 con 0x7f1294075d90 2024-09-15T02:58:24.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.667+0000 7f128ffff640 10 client.20405.objecter ms_dispatch 0x7f12a8000b90 command_reply(tid 1: 0 ) v1 2024-09-15T02:58:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.667+0000 7f128ffff640 10 client.20405.objecter _finish_command 1 = system:0 2024-09-15T02:58:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.667+0000 7f128ffff640 20 client.20405.objecter put_session s=0x7f12940756e0 osd=7 4 2024-09-15T02:58:24.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.667+0000 7f128ffff640 15 client.20405.objecter _session_command_op_remove 7 1 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 10 client.20405.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 15 client.20405.objecter _session_command_op_assign -1 2 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 20 client.20405.objecter _get_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 20 client.20405.objecter put_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 20 client.20405.objecter _get_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 15 client.20405.objecter _session_command_op_remove -1 2 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 20 client.20405.objecter get_session s=0x7f12940756e0 osd=7 3 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 15 client.20405.objecter _session_command_op_assign 7 2 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 20 client.20405.objecter put_session s=0x7f12940756e0 osd=7 4 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 10 client.20405.objecter _send_command 2 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 --> v2:172.21.15.104:6804/33347 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f12a813aa30 con 0x7f1294075d90 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f128ffff640 1 -- 172.21.15.53:0/2451223235 <== osd.7 v2:172.21.15.104:6804/33347 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f12a813aa30 con 0x7f1294075d90 2024-09-15T02:58:24.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f128ffff640 10 client.20405.objecter ms_dispatch 0x7f12a8000b90 command_reply(tid 2: 0 ) v1 2024-09-15T02:58:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f128ffff640 10 client.20405.objecter _finish_command 2 = system:0 2024-09-15T02:58:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f128ffff640 20 client.20405.objecter put_session s=0x7f12940756e0 osd=7 4 2024-09-15T02:58:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f128ffff640 15 client.20405.objecter _session_command_op_remove 7 2 2024-09-15T02:58:24.690 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-15T02:58:24.690 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:58:24.690 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 10 librados: watch_flush enter 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 10 librados: watch_flush exit 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 10 client.20405.objecter close_session for osd.7 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6804/33347 conn(0x7f1294075d90 msgr2=0x7f1294078220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6804/33347 conn(0x7f1294075d90 0x7f1294078220 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 20 client.20405.objecter put_session s=0x7f12940756e0 osd=7 1 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 20 client.20405.objecter shutdown clearing up homeless session... 2024-09-15T02:58:24.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 10 client.20405.objecter successfully canceled tick 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6816/32804 conn(0x7f129406e800 msgr2=0x7f1294070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.679+0000 7f12ae2d4640 1 --2- 172.21.15.53:0/2451223235 >> v2:172.21.15.104:6816/32804 conn(0x7f129406e800 0x7f1294070cb0 secure :-1 s=READY pgs=3705 cs=0 l=1 rev1=1 crypto rx=0x7f1298009850 tx=0x7f1298032000 comp rx=0 tx=0).stop 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 10 monclient: shutdown 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 msgr2=0x7f12a813a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 1 --2- 172.21.15.53:0/2451223235 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a80b2c40 0x7f12a813a000 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f12a000cbe0 tx=0x7f12a00019a0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ac871640 1 -- 172.21.15.53:0/2451223235 reap_dead start 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 shutdown_connections 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 >> 172.21.15.53:0/2451223235 conn(0x7f12a8063810 msgr2=0x7f12a806cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 shutdown_connections 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 1 -- 172.21.15.53:0/2451223235 wait complete. 2024-09-15T02:58:24.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.680+0000 7f12ae2d4640 1 librados: shutdown 2024-09-15T02:58:24.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-15T02:58:24.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.790+0000 7f6ba866f640 10 monclient: get_monmap_and_config 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.791+0000 7f6ba866f640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.791+0000 7f6ba866f640 10 monclient: monmap: 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:24.791796+0000 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:24.791796+0000 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:24.793 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.791+0000 7f6ba866f640 1 Processor -- start 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.791+0000 7f6ba866f640 1 -- start start 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.791+0000 7f6ba866f640 10 monclient: init 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ba00b0fa0 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6ba013b0c0 0x7f6ba013dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba013b0c0 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ba00a5920 0x7f6ba013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba00a5920 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00aed10 con 0x7f6ba00b0fa0 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba00a6b20 con 0x7f6ba00a5920 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba00b1370 con 0x7f6ba013b0c0 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba866f640 10 monclient(hunting): authenticate will time out at 1641.362496s 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ba00a5920 0x7f6ba013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ba00a5920 0x7f6ba013e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52564/0 (socket says 172.21.15.53:52564) 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 -- 172.21.15.53:0/1253426935 learned_addr learned my addr 172.21.15.53:0/1253426935 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): get_auth_request con 0x7f6ba00a5920 auth_method 0 2024-09-15T02:58:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 1 --2- 172.21.15.53:0/1253426935 >> v2:172.21.15.104:3300/0 conn(0x7f6ba013b0c0 0x7f6ba013dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 10 monclient(hunting): get_auth_request con 0x7f6ba013b0c0 auth_method 0 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba5c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_done global_id 26343 payload 274 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 -- 172.21.15.53:0/1253426935 >> v2:172.21.15.104:3300/0 conn(0x7f6ba013b0c0 msgr2=0x7f6ba013dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 --2- 172.21.15.53:0/1253426935 >> v2:172.21.15.104:3300/0 conn(0x7f6ba013b0c0 0x7f6ba013dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 -- 172.21.15.53:0/1253426935 >> v1:172.21.15.104:6789/0 conn(0x7f6ba00b0fa0 legacy=0x7f6ba013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 monclient: found mon.noname-a 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.792+0000 7f6ba6c0c640 1 -- 172.21.15.53:0/1253426935 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba00a6d20 con 0x7f6ba00a5920 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba6c0c640 1 --2- 172.21.15.53:0/1253426935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba00a5920 0x7f6ba013e270 secure :-1 s=READY pgs=5502 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00b1e60 tx=0x7f6b98009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98815335c2f91796 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 1 -- 172.21.15.53:0/1253426935 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98031400 con 0x7f6ba00a5920 2024-09-15T02:58:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: dump: 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 1 -- 172.21.15.53:0/1253426935 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9802d050 con 0x7f6ba00a5920 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba866f640 5 monclient: authenticate success, global_id 26343 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba866f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba866f640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 1 -- 172.21.15.53:0/1253426935 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b980317d0 con 0x7f6ba00a5920 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba866f640 4 set_mon_vals no callback set 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.793+0000 7f6ba5409640 10 monclient: dump: 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 10 monclient: shutdown 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 -- 172.21.15.53:0/1253426935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba00a5920 msgr2=0x7f6ba013e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1253426935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba00a5920 0x7f6ba013e270 secure :-1 s=READY pgs=5502 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00b1e60 tx=0x7f6b98009840 comp rx=0 tx=0).stop 2024-09-15T02:58:24.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 -- 172.21.15.53:0/1253426935 shutdown_connections 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1253426935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba00a5920 0x7f6ba013e270 unknown :-1 s=CLOSED pgs=5502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1253426935 >> v2:172.21.15.104:3300/0 conn(0x7f6ba013b0c0 0x7f6ba013dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 -- 172.21.15.53:0/1253426935 >> 172.21.15.53:0/1253426935 conn(0x7f6ba0062d90 msgr2=0x7f6ba0063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 -- 172.21.15.53:0/1253426935 shutdown_connections 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.794+0000 7f6ba866f640 1 -- 172.21.15.53:0/1253426935 wait complete. 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 10 monclient: build_initial_monmap 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 10 monclient: monmap: 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:24.809 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 1 librados: starting msgr at 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 1 librados: starting objecter 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 1 Processor -- start 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 1 -- start start 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 1 librados: setting wanted keys 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 1 librados: calling monclient init 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.795+0000 7f6ba866f640 10 monclient: init 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ba00a5920 0x7f6ba00a76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba00a5920 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ba00b0fa0 0x7f6ba00a7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba00b0fa0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba013b0c0 0x7f6ba0150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba013b0c0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00a7260 con 0x7f6ba013b0c0 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00a8170 con 0x7f6ba00b0fa0 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00a8f00 con 0x7f6ba00a5920 2024-09-15T02:58:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient(hunting): authenticate will time out at 1641.366193s 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba013b0c0 0x7f6ba0150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ba00a5920 0x7f6ba00a76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ba00a5920 0x7f6ba00a76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56468/0 (socket says 172.21.15.53:56468) 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba013b0c0 0x7f6ba0150850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52576/0 (socket says 172.21.15.53:52576) 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 1 -- 172.21.15.53:0/1148852292 learned_addr learned my addr 172.21.15.53:0/1148852292 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 10 monclient(hunting): get_auth_request con 0x7f6ba00a5920 auth_method 0 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): get_auth_request con 0x7f6ba013b0c0 auth_method 0 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ba00b0fa0 0x7f6ba00a7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 10 monclient(hunting): get_auth_request con 0x7f6ba00b0fa0 auth_method 0 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba640b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba5c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient(hunting): handle_auth_done global_id 26346 payload 824 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 -- 172.21.15.53:0/1148852292 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ba00a5920 msgr2=0x7f6ba00a76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ba00a5920 0x7f6ba00a76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 -- 172.21.15.53:0/1148852292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ba00b0fa0 msgr2=0x7f6ba00a7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ba00b0fa0 0x7f6ba00a7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient: _finish_hunting 0 2024-09-15T02:58:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 monclient: found mon.noname-a 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 -- 172.21.15.53:0/1148852292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b98009ae0 con 0x7f6ba013b0c0 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba6c0c640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba013b0c0 0x7f6ba0150850 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 crypto rx=0x7f6b98030000 tx=0x7f6b98034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 client.?.objecter ms_handle_connect 0x7f6ba013b0c0 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 1 -- 172.21.15.53:0/1148852292 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b980086a0 con 0x7f6ba013b0c0 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: dump: 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: _finish_auth 0 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 1 -- 172.21.15.53:0/1148852292 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b98008800 con 0x7f6ba013b0c0 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 5 monclient: authenticate success, global_id 26346 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient: _renew_subs 2024-09-15T02:58:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba013f160 con 0x7f6ba013b0c0 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 1 -- 172.21.15.53:0/1148852292 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98008bc0 con 0x7f6ba013b0c0 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6b967fc640 10 monclient: dump: 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.797+0000 7f6ba866f640 10 client.26346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.797+0000 7f6ba866f640 10 monclient: _renew_subs 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.797+0000 7f6ba866f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.797+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba0150fb0 con 0x7f6ba013b0c0 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.797+0000 7f6ba866f640 1 librados: init done 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.796+0000 7f6ba4c08640 4 set_mon_vals no callback set 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.797+0000 7f6ba866f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6ba866f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6ba866f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba00a73d0 con 0x7f6ba013b0c0 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6b967fc640 1 -- 172.21.15.53:0/1148852292 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f6b98038050 con 0x7f6ba013b0c0 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6b967fc640 1 --2- 172.21.15.53:0/1148852292 >> v2:172.21.15.104:6816/32804 conn(0x7f6b8406e800 0x7f6b84070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6b967fc640 1 -- 172.21.15.53:0/1148852292 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 502..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f6b980abff0 con 0x7f6ba013b0c0 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6b967fc640 10 client.26346.objecter ms_dispatch 0x7f6ba0000b90 osd_map(702..702 src has 502..702) v4 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6b967fc640 3 client.26346.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.798+0000 7f6b967fc640 3 client.26346.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.799+0000 7f6b967fc640 20 client.26346.objecter dump_active .. 0 homeless 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.799+0000 7f6ba640b640 1 --2- 172.21.15.53:0/1148852292 >> v2:172.21.15.104:6816/32804 conn(0x7f6b8406e800 0x7f6b84070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.799+0000 7f6ba640b640 10 monclient: get_auth_request con 0x7f6b8406e800 auth_method 0 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.799+0000 7f6b967fc640 10 client.26346.objecter ms_handle_connect 0x7f6b8406e800 2024-09-15T02:58:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.799+0000 7f6ba640b640 1 --2- 172.21.15.53:0/1148852292 >> v2:172.21.15.104:6816/32804 conn(0x7f6b8406e800 0x7f6b84070cb0 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c003aa0 tx=0x7f6b9c000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:24.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.802+0000 7f6b967fc640 1 -- 172.21.15.53:0/1148852292 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6b98076350 con 0x7f6ba013b0c0 2024-09-15T02:58:24.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.802+0000 7f6b967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:24.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.802+0000 7f6b967fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:24.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.967+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ba00a9070 con 0x7f6b8406e800 2024-09-15T02:58:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.968+0000 7f6b967fc640 1 -- 172.21.15.53:0/1148852292 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6ba00a9070 con 0x7f6b8406e800 2024-09-15T02:58:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 7 to deep-scrub 2024-09-15T02:58:24.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.969+0000 7f6ba866f640 10 librados: watch_flush enter 2024-09-15T02:58:24.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 10 librados: watch_flush exit 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 20 client.26346.objecter shutdown clearing up homeless session... 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 10 client.26346.objecter successfully canceled tick 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 >> v2:172.21.15.104:6816/32804 conn(0x7f6b8406e800 msgr2=0x7f6b84070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1148852292 >> v2:172.21.15.104:6816/32804 conn(0x7f6b8406e800 0x7f6b84070cb0 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c003aa0 tx=0x7f6b9c000bc0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 10 monclient: shutdown 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba013b0c0 msgr2=0x7f6ba0150850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba013b0c0 0x7f6ba0150850 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 crypto rx=0x7f6b98030000 tx=0x7f6b98034000 comp rx=0 tx=0).stop 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 shutdown_connections 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba013b0c0 0x7f6ba0150850 unknown :-1 s=CLOSED pgs=5503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1148852292 >> v2:172.21.15.104:6816/32804 conn(0x7f6b8406e800 0x7f6b84070cb0 unknown :-1 s=CLOSED pgs=3706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ba00b0fa0 0x7f6ba00a7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 --2- 172.21.15.53:0/1148852292 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ba00a5920 0x7f6ba00a76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:24.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 >> 172.21.15.53:0/1148852292 conn(0x7f6ba0062d90 msgr2=0x7f6ba00b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:24.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 shutdown_connections 2024-09-15T02:58:24.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 -- 172.21.15.53:0/1148852292 wait complete. 2024-09-15T02:58:24.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:24.970+0000 7f6ba866f640 1 librados: shutdown 2024-09-15T02:58:24.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:58:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.076+0000 7f74674b8640 10 monclient: get_monmap_and_config 2024-09-15T02:58:25.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.076+0000 7f74674b8640 10 monclient: build_initial_monmap 2024-09-15T02:58:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.076+0000 7f74674b8640 10 monclient: monmap: 2024-09-15T02:58:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:25.077681+0000 2024-09-15T02:58:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:25.077681+0000 2024-09-15T02:58:25.079 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:25.080 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:25.081 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.077+0000 7f74674b8640 1 Processor -- start 2024-09-15T02:58:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.077+0000 7f74674b8640 1 -- start start 2024-09-15T02:58:25.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.077+0000 7f74674b8640 10 monclient: init 2024-09-15T02:58:25.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T02:58:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b0e80 0x7f74600b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient(hunting): picked mon.noname-b con 0x7f74600b0e80 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f74600b1250 0x7f746013def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient(hunting): picked mon.noname-c con 0x7f74600b1250 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f746013e430 addr v1:172.21.15.53:6789/0 2024-09-15T02:58:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746013da60 con 0x7f746013e430 2024-09-15T02:58:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74600b0990 con 0x7f74600b1250 2024-09-15T02:58:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:25.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746013e8b0 con 0x7f74600b0e80 2024-09-15T02:58:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f74674b8640 10 monclient(hunting): authenticate will time out at 1641.648486s 2024-09-15T02:58:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b0e80 0x7f74600b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b0e80 0x7f74600b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56478/0 (socket says 172.21.15.53:56478) 2024-09-15T02:58:25.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 -- 172.21.15.53:0/1994087111 learned_addr learned my addr 172.21.15.53:0/1994087111 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): get_auth_request con 0x7f74600b0e80 auth_method 0 2024-09-15T02:58:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465a55640 1 --1- 172.21.15.53:0/1994087111 >> v1:172.21.15.53:6789/0 conn(0x7f746013e430 0x7f74600af3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59294/0 (socket says 172.21.15.53:59294) 2024-09-15T02:58:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:25.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 1 --2- 172.21.15.53:0/1994087111 >> v2:172.21.15.104:3300/0 conn(0x7f74600b1250 0x7f746013def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 10 monclient(hunting): get_auth_request con 0x7f74600b1250 auth_method 0 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7464a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:25.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient(hunting): handle_auth_done global_id 20411 payload 274 2024-09-15T02:58:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7457fff640 1 -- 172.21.15.53:0/1994087111 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591079366 0 0) 0x7f746013da60 con 0x7f746013e430 2024-09-15T02:58:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 -- 172.21.15.53:0/1994087111 >> v2:172.21.15.104:3300/0 conn(0x7f74600b1250 msgr2=0x7f746013def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 --2- 172.21.15.53:0/1994087111 >> v2:172.21.15.104:3300/0 conn(0x7f74600b1250 0x7f746013def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 -- 172.21.15.53:0/1994087111 >> v1:172.21.15.53:6789/0 conn(0x7f746013e430 legacy=0x7f74600af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient: _finish_hunting 0 2024-09-15T02:58:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 monclient: found mon.noname-b 2024-09-15T02:58:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7465254640 1 -- 172.21.15.53:0/1994087111 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74600a6a30 con 0x7f74600b0e80 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.078+0000 7f7457fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7465254640 1 --2- 172.21.15.53:0/1994087111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b0e80 0x7f74600b0450 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7f745c009910 tx=0x7f745c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e281e6a5f29fcee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 1 -- 172.21.15.53:0/1994087111 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c02adb0 con 0x7f74600b0e80 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: dump: 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.102 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: _finish_auth 0 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f74674b8640 5 monclient: authenticate success, global_id 20411 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f74674b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 1 -- 172.21.15.53:0/1994087111 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745c032440 con 0x7f74600b0e80 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f74674b8640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f74674b8640 4 set_mon_vals no callback set 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 1 -- 172.21.15.53:0/1994087111 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c0327e0 con 0x7f74600b0e80 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.079+0000 7f7457fff640 10 monclient: dump: 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:25.104 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 10 monclient: shutdown 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 -- 172.21.15.53:0/1994087111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b0e80 msgr2=0x7f74600b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 --2- 172.21.15.53:0/1994087111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b0e80 0x7f74600b0450 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7f745c009910 tx=0x7f745c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 -- 172.21.15.53:0/1994087111 shutdown_connections 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 --2- 172.21.15.53:0/1994087111 >> v2:172.21.15.104:3300/0 conn(0x7f74600b1250 0x7f746013def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 --2- 172.21.15.53:0/1994087111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b0e80 0x7f74600b0450 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 -- 172.21.15.53:0/1994087111 >> 172.21.15.53:0/1994087111 conn(0x7f7460062d90 msgr2=0x7f7460063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:25.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 -- 172.21.15.53:0/1994087111 shutdown_connections 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.080+0000 7f74674b8640 1 -- 172.21.15.53:0/1994087111 wait complete. 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 10 monclient: build_initial_monmap 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 10 monclient: monmap: 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:25.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 1 librados: starting msgr at 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 1 librados: starting objecter 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 1 Processor -- start 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 1 -- start start 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 1 librados: setting wanted keys 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 1 librados: calling monclient init 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.081+0000 7f74674b8640 10 monclient: init 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74600b0e80 0x7f74600aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): picked mon.noname-b con 0x7f74600b0e80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b1250 0x7f74600a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): picked mon.noname-c con 0x7f74600b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:25.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f746013e430 0x7f746014d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): picked mon.noname-a con 0x7f746013e430 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746014d170 con 0x7f746013e430 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746014d450 con 0x7f74600b0e80 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74600ab120 con 0x7f74600b1250 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74674b8640 10 monclient(hunting): authenticate will time out at 1641.652263s 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b1250 0x7f74600a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b1250 0x7f74600a9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:56486/0 (socket says 172.21.15.53:56486) 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f746013e430 0x7f746014d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 -- 172.21.15.53:0/1509424890 learned_addr learned my addr 172.21.15.53:0/1509424890 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:25.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): get_auth_request con 0x7f74600b1250 auth_method 0 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 10 monclient(hunting): get_auth_request con 0x7f746013e430 auth_method 0 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:25.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465254640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74600b0e80 0x7f74600aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465254640 10 monclient(hunting): get_auth_request con 0x7f74600b0e80 auth_method 0 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465254640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465254640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient(hunting): handle_auth_done global_id 20414 payload 824 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 -- 172.21.15.53:0/1509424890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74600b0e80 msgr2=0x7f74600aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74600b0e80 0x7f74600aabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 -- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f746013e430 msgr2=0x7f746014d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f746013e430 0x7f746014d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient: _finish_hunting 0 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 monclient: found mon.noname-c 2024-09-15T02:58:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.53:3301/0 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 -- 172.21.15.53:0/1509424890 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c009450 con 0x7f74600b1250 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7465a55640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f746013e430 0x7f746014d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f7464a53640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b1250 0x7f74600a9de0 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f745000ea50 tx=0x7f74500090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74557fa640 10 client.?.objecter ms_handle_connect 0x7f74600b1250 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74557fa640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.082+0000 7f74557fa640 1 -- 172.21.15.53:0/1509424890 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745000ca10 con 0x7f74600b1250 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: dump: 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.111 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: _finish_auth 0 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 5 monclient: authenticate success, global_id 20414 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 1 -- 172.21.15.53:0/1509424890 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745000cb70 con 0x7f74600b1250 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 1 -- 172.21.15.53:0/1509424890 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450013470 con 0x7f74600b1250 2024-09-15T02:58:25.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 10 monclient: _renew_subs 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74600aa320 con 0x7f74600b1250 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74557fa640 10 monclient: dump: 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:25.113 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74577fe640 4 set_mon_vals no callback set 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 10 client.20414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 10 monclient: _renew_subs 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746014e040 con 0x7f74600b1250 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.083+0000 7f74674b8640 1 librados: init done 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.084+0000 7f74674b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.084+0000 7f74674b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.084+0000 7f74674b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.53:3301/0 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.084+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74600aa540 con 0x7f74600b1250 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.084+0000 7f74557fa640 1 -- 172.21.15.53:0/1509424890 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f7450013860 con 0x7f74600b1250 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.084+0000 7f74557fa640 1 --2- 172.21.15.53:0/1509424890 >> v2:172.21.15.104:6816/32804 conn(0x7f744406e800 0x7f7444070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:25.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.084+0000 7f74557fa640 1 -- 172.21.15.53:0/1509424890 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(702..702 src has 502..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f745008be30 con 0x7f74600b1250 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f74557fa640 10 client.20414.objecter ms_dispatch 0x7f7460000b90 osd_map(702..702 src has 502..702) v4 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f74557fa640 3 client.20414.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f74557fa640 3 client.20414.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f74557fa640 20 client.20414.objecter dump_active .. 0 homeless 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f7465254640 1 --2- 172.21.15.53:0/1509424890 >> v2:172.21.15.104:6816/32804 conn(0x7f744406e800 0x7f7444070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f7465254640 10 monclient: get_auth_request con 0x7f744406e800 auth_method 0 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f74557fa640 10 client.20414.objecter ms_handle_connect 0x7f744406e800 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.085+0000 7f7465254640 1 --2- 172.21.15.53:0/1509424890 >> v2:172.21.15.104:6816/32804 conn(0x7f744406e800 0x7f7444070cb0 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f745c009910 tx=0x7f745c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.088+0000 7f74557fa640 1 -- 172.21.15.53:0/1509424890 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f74500560d0 con 0x7f74600b1250 2024-09-15T02:58:25.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.088+0000 7f74557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:25.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.088+0000 7f74557fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:25.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.247+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7460063ff0 con 0x7f744406e800 2024-09-15T02:58:25.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.252+0000 7f74557fa640 1 -- 172.21.15.53:0/1509424890 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+153536 (secure 0 0 0) 0x7f7460063ff0 con 0x7f744406e800 2024-09-15T02:58:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:58:25.255 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:58:25.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 10 librados: watch_flush enter 2024-09-15T02:58:25.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 10 librados: watch_flush exit 2024-09-15T02:58:25.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 20 client.20414.objecter shutdown clearing up homeless session... 2024-09-15T02:58:25.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 10 client.20414.objecter successfully canceled tick 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 >> v2:172.21.15.104:6816/32804 conn(0x7f744406e800 msgr2=0x7f7444070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 --2- 172.21.15.53:0/1509424890 >> v2:172.21.15.104:6816/32804 conn(0x7f744406e800 0x7f7444070cb0 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f745c009910 tx=0x7f745c032000 comp rx=0 tx=0).stop 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 10 monclient: shutdown 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b1250 msgr2=0x7f74600a9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b1250 0x7f74600a9de0 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f745000ea50 tx=0x7f74500090f0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 shutdown_connections 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f746013e430 0x7f746014d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f74600b1250 0x7f74600a9de0 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 --2- 172.21.15.53:0/1509424890 >> v2:172.21.15.104:6816/32804 conn(0x7f744406e800 0x7f7444070cb0 unknown :-1 s=CLOSED pgs=3707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 --2- 172.21.15.53:0/1509424890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74600b0e80 0x7f74600aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:25.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 >> 172.21.15.53:0/1509424890 conn(0x7f7460062d90 msgr2=0x7f74600af190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:25.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 shutdown_connections 2024-09-15T02:58:25.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 -- 172.21.15.53:0/1509424890 wait complete. 2024-09-15T02:58:25.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:25.255+0000 7f74674b8640 1 librados: shutdown 2024-09-15T02:58:25.267 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":1147,"stamp":"2024-09-15T02:58:23.540950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2471416,"kb_used_data":50928,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752503304,"statfs":{"total":773094113280,"available":770563383296,"internally_reserved":0,"allocated":52150272,"data_stored":25972442,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.004202"},"pg_stats":[{"pgid":"12.10","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539924+0000","last_change":"2024-09-15T02:44:21.592595+0000","last_active":"2024-09-15T02:58:16.539924+0000","last_peered":"2024-09-15T02:58:16.539924+0000","last_clean":"2024-09-15T02:58:16.539924+0000","last_became_active":"2024-09-15T02:44:21.589224+0000","last_became_peered":"2024-09-15T02:44:21.589224+0000","last_unstale":"2024-09-15T02:58:16.539924+0000","last_undegraded":"2024-09-15T02:58:16.539924+0000","last_fullsized":"2024-09-15T02:58:16.539924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:30:34.422407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836595+0000","last_change":"2024-09-15T02:44:21.472953+0000","last_active":"2024-09-15T02:58:18.836595+0000","last_peered":"2024-09-15T02:58:18.836595+0000","last_clean":"2024-09-15T02:58:18.836595+0000","last_became_active":"2024-09-15T02:44:21.472608+0000","last_became_peered":"2024-09-15T02:44:21.472608+0000","last_unstale":"2024-09-15T02:58:18.836595+0000","last_undegraded":"2024-09-15T02:58:18.836595+0000","last_fullsized":"2024-09-15T02:58:18.836595+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:48.488509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":657,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.335994+0000","last_change":"2024-09-15T02:44:21.605637+0000","last_active":"2024-09-15T02:58:18.335994+0000","last_peered":"2024-09-15T02:58:18.335994+0000","last_clean":"2024-09-15T02:58:18.335994+0000","last_became_active":"2024-09-15T02:44:21.604946+0000","last_became_peered":"2024-09-15T02:44:21.604946+0000","last_unstale":"2024-09-15T02:58:18.335994+0000","last_undegraded":"2024-09-15T02:58:18.335994+0000","last_fullsized":"2024-09-15T02:58:18.335994+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:21:30.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":647,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:19.570825+0000","last_change":"2024-09-15T02:44:21.631801+0000","last_active":"2024-09-15T02:58:19.570825+0000","last_peered":"2024-09-15T02:58:19.570825+0000","last_clean":"2024-09-15T02:58:19.570825+0000","last_became_active":"2024-09-15T02:44:21.621686+0000","last_became_peered":"2024-09-15T02:44:21.621686+0000","last_unstale":"2024-09-15T02:58:19.570825+0000","last_undegraded":"2024-09-15T02:58:19.570825+0000","last_fullsized":"2024-09-15T02:58:19.570825+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":645,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836680+0000","last_change":"2024-09-15T02:44:21.559633+0000","last_active":"2024-09-15T02:58:18.836680+0000","last_peered":"2024-09-15T02:58:18.836680+0000","last_clean":"2024-09-15T02:58:18.836680+0000","last_became_active":"2024-09-15T02:44:21.558951+0000","last_became_peered":"2024-09-15T02:44:21.558951+0000","last_unstale":"2024-09-15T02:58:18.836680+0000","last_undegraded":"2024-09-15T02:58:18.836680+0000","last_fullsized":"2024-09-15T02:58:18.836680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:27.075112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236232+0000","last_change":"2024-09-15T02:44:21.593138+0000","last_active":"2024-09-15T02:58:17.236232+0000","last_peered":"2024-09-15T02:58:17.236232+0000","last_clean":"2024-09-15T02:58:17.236232+0000","last_became_active":"2024-09-15T02:44:21.592665+0000","last_became_peered":"2024-09-15T02:44:21.592665+0000","last_unstale":"2024-09-15T02:58:17.236232+0000","last_undegraded":"2024-09-15T02:58:17.236232+0000","last_fullsized":"2024-09-15T02:58:17.236232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:20.534135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235717+0000","last_change":"2024-09-15T02:44:21.537510+0000","last_active":"2024-09-15T02:58:17.235717+0000","last_peered":"2024-09-15T02:58:17.235717+0000","last_clean":"2024-09-15T02:58:17.235717+0000","last_became_active":"2024-09-15T02:44:21.537279+0000","last_became_peered":"2024-09-15T02:44:21.537279+0000","last_unstale":"2024-09-15T02:58:17.235717+0000","last_undegraded":"2024-09-15T02:58:17.235717+0000","last_fullsized":"2024-09-15T02:58:17.235717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:55:56.290343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235427+0000","last_change":"2024-09-15T02:44:21.613091+0000","last_active":"2024-09-15T02:58:17.235427+0000","last_peered":"2024-09-15T02:58:17.235427+0000","last_clean":"2024-09-15T02:58:17.235427+0000","last_became_active":"2024-09-15T02:44:21.612841+0000","last_became_peered":"2024-09-15T02:44:21.612841+0000","last_unstale":"2024-09-15T02:58:17.235427+0000","last_undegraded":"2024-09-15T02:58:17.235427+0000","last_fullsized":"2024-09-15T02:58:17.235427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:28:24.283127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":647,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:19.570371+0000","last_change":"2024-09-15T02:44:21.543344+0000","last_active":"2024-09-15T02:58:19.570371+0000","last_peered":"2024-09-15T02:58:19.570371+0000","last_clean":"2024-09-15T02:58:19.570371+0000","last_became_active":"2024-09-15T02:44:21.522860+0000","last_became_peered":"2024-09-15T02:44:21.522860+0000","last_unstale":"2024-09-15T02:58:19.570371+0000","last_undegraded":"2024-09-15T02:58:19.570371+0000","last_fullsized":"2024-09-15T02:58:19.570371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541359+0000","last_change":"2024-09-15T02:44:21.538578+0000","last_active":"2024-09-15T02:58:16.541359+0000","last_peered":"2024-09-15T02:58:16.541359+0000","last_clean":"2024-09-15T02:58:16.541359+0000","last_became_active":"2024-09-15T02:44:21.535334+0000","last_became_peered":"2024-09-15T02:44:21.535334+0000","last_unstale":"2024-09-15T02:58:16.541359+0000","last_undegraded":"2024-09-15T02:58:16.541359+0000","last_fullsized":"2024-09-15T02:58:16.541359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:38:25.943125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542872+0000","last_change":"2024-09-15T02:44:21.614405+0000","last_active":"2024-09-15T02:58:17.542872+0000","last_peered":"2024-09-15T02:58:17.542872+0000","last_clean":"2024-09-15T02:58:17.542872+0000","last_became_active":"2024-09-15T02:44:21.614186+0000","last_became_peered":"2024-09-15T02:44:21.614186+0000","last_unstale":"2024-09-15T02:58:17.542872+0000","last_undegraded":"2024-09-15T02:58:17.542872+0000","last_fullsized":"2024-09-15T02:58:17.542872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:52.698525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542738+0000","last_change":"2024-09-15T02:44:21.527859+0000","last_active":"2024-09-15T02:58:17.542738+0000","last_peered":"2024-09-15T02:58:17.542738+0000","last_clean":"2024-09-15T02:58:17.542738+0000","last_became_active":"2024-09-15T02:44:21.527331+0000","last_became_peered":"2024-09-15T02:44:21.527331+0000","last_unstale":"2024-09-15T02:58:17.542738+0000","last_undegraded":"2024-09-15T02:58:17.542738+0000","last_fullsized":"2024-09-15T02:58:17.542738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:55:15.776872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":653,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235679+0000","last_change":"2024-09-15T02:43:23.626724+0000","last_active":"2024-09-15T02:58:17.235679+0000","last_peered":"2024-09-15T02:58:17.235679+0000","last_clean":"2024-09-15T02:58:17.235679+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:58:17.235679+0000","last_undegraded":"2024-09-15T02:58:17.235679+0000","last_fullsized":"2024-09-15T02:58:17.235679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:48.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540373+0000","last_change":"2024-09-15T02:44:21.539798+0000","last_active":"2024-09-15T02:58:17.540373+0000","last_peered":"2024-09-15T02:58:17.540373+0000","last_clean":"2024-09-15T02:58:17.540373+0000","last_became_active":"2024-09-15T02:44:21.530274+0000","last_became_peered":"2024-09-15T02:44:21.530274+0000","last_unstale":"2024-09-15T02:58:17.540373+0000","last_undegraded":"2024-09-15T02:58:17.540373+0000","last_fullsized":"2024-09-15T02:58:17.540373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:24:08.051420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:19.570532+0000","last_change":"2024-09-15T02:43:23.426048+0000","last_active":"2024-09-15T02:58:19.570532+0000","last_peered":"2024-09-15T02:58:19.570532+0000","last_clean":"2024-09-15T02:58:19.570532+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:58:19.570532+0000","last_undegraded":"2024-09-15T02:58:19.570532+0000","last_fullsized":"2024-09-15T02:58:19.570532+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":657,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.335845+0000","last_change":"2024-09-15T02:44:21.579868+0000","last_active":"2024-09-15T02:58:18.335845+0000","last_peered":"2024-09-15T02:58:18.335845+0000","last_clean":"2024-09-15T02:58:18.335845+0000","last_became_active":"2024-09-15T02:44:21.579154+0000","last_became_peered":"2024-09-15T02:44:21.579154+0000","last_unstale":"2024-09-15T02:58:18.335845+0000","last_undegraded":"2024-09-15T02:58:18.335845+0000","last_fullsized":"2024-09-15T02:58:18.335845+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:45:09.149484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":650,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914260+0000","last_change":"2024-09-15T02:43:23.420483+0000","last_active":"2024-09-15T02:58:16.914260+0000","last_peered":"2024-09-15T02:58:16.914260+0000","last_clean":"2024-09-15T02:58:16.914260+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:58:16.914260+0000","last_undegraded":"2024-09-15T02:58:16.914260+0000","last_fullsized":"2024-09-15T02:58:16.914260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:03:14.233618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":657,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.335964+0000","last_change":"2024-09-15T02:44:21.537181+0000","last_active":"2024-09-15T02:58:18.335964+0000","last_peered":"2024-09-15T02:58:18.335964+0000","last_clean":"2024-09-15T02:58:18.335964+0000","last_became_active":"2024-09-15T02:44:21.533501+0000","last_became_peered":"2024-09-15T02:44:21.533501+0000","last_unstale":"2024-09-15T02:58:18.335964+0000","last_undegraded":"2024-09-15T02:58:18.335964+0000","last_fullsized":"2024-09-15T02:58:18.335964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:42.258617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:19.570704+0000","last_change":"2024-09-15T02:43:23.426071+0000","last_active":"2024-09-15T02:58:19.570704+0000","last_peered":"2024-09-15T02:58:19.570704+0000","last_clean":"2024-09-15T02:58:19.570704+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:58:19.570704+0000","last_undegraded":"2024-09-15T02:58:19.570704+0000","last_fullsized":"2024-09-15T02:58:19.570704+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":652,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540360+0000","last_change":"2024-09-15T02:44:21.561913+0000","last_active":"2024-09-15T02:58:17.540360+0000","last_peered":"2024-09-15T02:58:17.540360+0000","last_clean":"2024-09-15T02:58:17.540360+0000","last_became_active":"2024-09-15T02:44:21.560425+0000","last_became_peered":"2024-09-15T02:44:21.560425+0000","last_unstale":"2024-09-15T02:58:17.540360+0000","last_undegraded":"2024-09-15T02:58:17.540360+0000","last_fullsized":"2024-09-15T02:58:17.540360+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:40:57.301643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":658,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836590+0000","last_change":"2024-09-15T02:43:23.423983+0000","last_active":"2024-09-15T02:58:18.836590+0000","last_peered":"2024-09-15T02:58:18.836590+0000","last_clean":"2024-09-15T02:58:18.836590+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:58:18.836590+0000","last_undegraded":"2024-09-15T02:58:18.836590+0000","last_fullsized":"2024-09-15T02:58:18.836590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:45.683030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539798+0000","last_change":"2024-09-15T02:44:21.500932+0000","last_active":"2024-09-15T02:58:16.539798+0000","last_peered":"2024-09-15T02:58:16.539798+0000","last_clean":"2024-09-15T02:58:16.539798+0000","last_became_active":"2024-09-15T02:44:21.500646+0000","last_became_peered":"2024-09-15T02:44:21.500646+0000","last_unstale":"2024-09-15T02:58:16.539798+0000","last_undegraded":"2024-09-15T02:58:16.539798+0000","last_fullsized":"2024-09-15T02:58:16.539798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:08.733289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":663,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.540358+0000","last_change":"2024-09-15T02:43:23.419938+0000","last_active":"2024-09-15T02:58:17.540358+0000","last_peered":"2024-09-15T02:58:17.540358+0000","last_clean":"2024-09-15T02:58:17.540358+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:58:17.540358+0000","last_undegraded":"2024-09-15T02:58:17.540358+0000","last_fullsized":"2024-09-15T02:58:17.540358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:09:35.753407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235268+0000","last_change":"2024-09-15T02:44:21.504909+0000","last_active":"2024-09-15T02:58:17.235268+0000","last_peered":"2024-09-15T02:58:17.235268+0000","last_clean":"2024-09-15T02:58:17.235268+0000","last_became_active":"2024-09-15T02:44:21.504665+0000","last_became_peered":"2024-09-15T02:44:21.504665+0000","last_unstale":"2024-09-15T02:58:17.235268+0000","last_undegraded":"2024-09-15T02:58:17.235268+0000","last_fullsized":"2024-09-15T02:58:17.235268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:17:58.326032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":649,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913924+0000","last_change":"2024-09-15T02:43:23.420430+0000","last_active":"2024-09-15T02:58:16.913924+0000","last_peered":"2024-09-15T02:58:16.913924+0000","last_clean":"2024-09-15T02:58:16.913924+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:58:16.913924+0000","last_undegraded":"2024-09-15T02:58:16.913924+0000","last_fullsized":"2024-09-15T02:58:16.913924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:05:17.337850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.543353+0000","last_change":"2024-09-15T02:44:21.645181+0000","last_active":"2024-09-15T02:58:16.543353+0000","last_peered":"2024-09-15T02:58:16.543353+0000","last_clean":"2024-09-15T02:58:16.543353+0000","last_became_active":"2024-09-15T02:44:21.644939+0000","last_became_peered":"2024-09-15T02:44:21.644939+0000","last_unstale":"2024-09-15T02:58:16.543353+0000","last_undegraded":"2024-09-15T02:58:16.543353+0000","last_fullsized":"2024-09-15T02:58:16.543353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:22:20.773021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":657,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:18.836538+0000","last_change":"2024-09-15T02:43:23.423477+0000","last_active":"2024-09-15T02:58:18.836538+0000","last_peered":"2024-09-15T02:58:18.836538+0000","last_clean":"2024-09-15T02:58:18.836538+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:58:18.836538+0000","last_undegraded":"2024-09-15T02:58:18.836538+0000","last_fullsized":"2024-09-15T02:58:18.836538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:20.381248+0000","last_clean_scrub_stamp":"2024-09-15T02:43:20.381248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:05:30.605263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":735,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.913790+0000","last_change":"2024-09-15T02:43:20.387731+0000","last_active":"2024-09-15T02:58:16.913790+0000","last_peered":"2024-09-15T02:58:16.913790+0000","last_clean":"2024-09-15T02:58:16.913790+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:58:16.913790+0000","last_undegraded":"2024-09-15T02:58:16.913790+0000","last_fullsized":"2024-09-15T02:58:16.913790+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:43:19.374225+0000","last_clean_scrub_stamp":"2024-09-15T02:43:19.374225+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:33:29.816927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":639,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.914166+0000","last_change":"2024-09-15T02:44:21.629491+0000","last_active":"2024-09-15T02:58:16.914166+0000","last_peered":"2024-09-15T02:58:16.914166+0000","last_clean":"2024-09-15T02:58:16.914166+0000","last_became_active":"2024-09-15T02:44:21.629239+0000","last_became_peered":"2024-09-15T02:44:21.629239+0000","last_unstale":"2024-09-15T02:58:16.914166+0000","last_undegraded":"2024-09-15T02:58:16.914166+0000","last_fullsized":"2024-09-15T02:58:16.914166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:56.354264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":647,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:19.570789+0000","last_change":"2024-09-15T02:44:21.573593+0000","last_active":"2024-09-15T02:58:19.570789+0000","last_peered":"2024-09-15T02:58:19.570789+0000","last_clean":"2024-09-15T02:58:19.570789+0000","last_became_active":"2024-09-15T02:44:21.571415+0000","last_became_peered":"2024-09-15T02:44:21.571415+0000","last_unstale":"2024-09-15T02:58:19.570789+0000","last_undegraded":"2024-09-15T02:58:19.570789+0000","last_fullsized":"2024-09-15T02:58:19.570789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":647,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:19.570600+0000","last_change":"2024-09-15T02:44:21.629604+0000","last_active":"2024-09-15T02:58:19.570600+0000","last_peered":"2024-09-15T02:58:19.570600+0000","last_clean":"2024-09-15T02:58:19.570600+0000","last_became_active":"2024-09-15T02:44:21.628817+0000","last_became_peered":"2024-09-15T02:44:21.628817+0000","last_unstale":"2024-09-15T02:58:19.570600+0000","last_undegraded":"2024-09-15T02:58:19.570600+0000","last_fullsized":"2024-09-15T02:58:19.570600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.236078+0000","last_change":"2024-09-15T02:44:21.583579+0000","last_active":"2024-09-15T02:58:17.236078+0000","last_peered":"2024-09-15T02:58:17.236078+0000","last_clean":"2024-09-15T02:58:17.236078+0000","last_became_active":"2024-09-15T02:44:21.583340+0000","last_became_peered":"2024-09-15T02:44:21.583340+0000","last_unstale":"2024-09-15T02:58:17.236078+0000","last_undegraded":"2024-09-15T02:58:17.236078+0000","last_fullsized":"2024-09-15T02:58:17.236078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:01:58.119370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235646+0000","last_change":"2024-09-15T02:44:21.616190+0000","last_active":"2024-09-15T02:58:17.235646+0000","last_peered":"2024-09-15T02:58:17.235646+0000","last_clean":"2024-09-15T02:58:17.235646+0000","last_became_active":"2024-09-15T02:44:21.615991+0000","last_became_peered":"2024-09-15T02:44:21.615991+0000","last_unstale":"2024-09-15T02:58:17.235646+0000","last_undegraded":"2024-09-15T02:58:17.235646+0000","last_fullsized":"2024-09-15T02:58:17.235646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:25:27.900125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.539926+0000","last_change":"2024-09-15T02:44:21.608529+0000","last_active":"2024-09-15T02:58:16.539926+0000","last_peered":"2024-09-15T02:58:16.539926+0000","last_clean":"2024-09-15T02:58:16.539926+0000","last_became_active":"2024-09-15T02:44:21.588781+0000","last_became_peered":"2024-09-15T02:44:21.588781+0000","last_unstale":"2024-09-15T02:58:16.539926+0000","last_undegraded":"2024-09-15T02:58:16.539926+0000","last_fullsized":"2024-09-15T02:58:16.539926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:49.754802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.614367+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.613899+0000","last_became_peered":"2024-09-15T02:44:21.613899+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:18:47.868539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1","version":"22'7","reported_seq":655,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235332+0000","last_change":"2024-09-15T02:44:21.595390+0000","last_active":"2024-09-15T02:58:17.235332+0000","last_peered":"2024-09-15T02:58:17.235332+0000","last_clean":"2024-09-15T02:58:17.235332+0000","last_became_active":"2024-09-15T02:44:21.594670+0000","last_became_peered":"2024-09-15T02:44:21.594670+0000","last_unstale":"2024-09-15T02:58:17.235332+0000","last_undegraded":"2024-09-15T02:58:17.235332+0000","last_fullsized":"2024-09-15T02:58:17.235332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:52:07.118143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542731+0000","last_change":"2024-09-15T02:44:21.536146+0000","last_active":"2024-09-15T02:58:17.542731+0000","last_peered":"2024-09-15T02:58:17.542731+0000","last_clean":"2024-09-15T02:58:17.542731+0000","last_became_active":"2024-09-15T02:44:21.535920+0000","last_became_peered":"2024-09-15T02:44:21.535920+0000","last_unstale":"2024-09-15T02:58:17.542731+0000","last_undegraded":"2024-09-15T02:58:17.542731+0000","last_fullsized":"2024-09-15T02:58:17.542731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.095882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":642,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.235267+0000","last_change":"2024-09-15T02:44:21.578890+0000","last_active":"2024-09-15T02:58:17.235267+0000","last_peered":"2024-09-15T02:58:17.235267+0000","last_clean":"2024-09-15T02:58:17.235267+0000","last_became_active":"2024-09-15T02:44:21.578571+0000","last_became_peered":"2024-09-15T02:44:21.578571+0000","last_unstale":"2024-09-15T02:58:17.235267+0000","last_undegraded":"2024-09-15T02:58:17.235267+0000","last_fullsized":"2024-09-15T02:58:17.235267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:01.395702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":646,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-09-15T02:58:17.542928+0000","last_change":"2024-09-15T02:44:21.556333+0000","last_active":"2024-09-15T02:58:17.542928+0000","last_peered":"2024-09-15T02:58:17.542928+0000","last_clean":"2024-09-15T02:58:17.542928+0000","last_became_active":"2024-09-15T02:44:21.556078+0000","last_became_peered":"2024-09-15T02:44:21.556078+0000","last_unstale":"2024-09-15T02:58:17.542928+0000","last_undegraded":"2024-09-15T02:58:17.542928+0000","last_fullsized":"2024-09-15T02:58:17.542928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:35:40.520618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.541145+0000","last_change":"2024-09-15T02:44:21.583194+0000","last_active":"2024-09-15T02:58:16.541145+0000","last_peered":"2024-09-15T02:58:16.541145+0000","last_clean":"2024-09-15T02:58:16.541145+0000","last_became_active":"2024-09-15T02:44:21.582718+0000","last_became_peered":"2024-09-15T02:44:21.582718+0000","last_unstale":"2024-09-15T02:58:16.541145+0000","last_undegraded":"2024-09-15T02:58:16.541145+0000","last_fullsized":"2024-09-15T02:58:16.541145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:48:37.528157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":646,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-15T02:58:16.540867+0000","last_change":"2024-09-15T02:44:21.584252+0000","last_active":"2024-09-15T02:58:16.540867+0000","last_peered":"2024-09-15T02:58:16.540867+0000","last_clean":"2024-09-15T02:58:16.540867+0000","last_became_active":"2024-09-15T02:44:21.584028+0000","last_became_peered":"2024-09-15T02:44:21.584028+0000","last_unstale":"2024-09-15T02:58:16.540867+0000","last_undegraded":"2024-09-15T02:58:16.540867+0000","last_fullsized":"2024-09-15T02:58:16.540867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:14.221941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575031,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312068,"kb_used_data":6388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059772,"statfs":{"total":96636764160,"available":96317206528,"internally_reserved":0,"allocated":6541312,"data_stored":3635868,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.69399999999999995,"5min":0.75,"15min":1.3959999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.28100000000000003},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.80500000000000005},"last":0.76500000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.55300000000000005},"min":{"1min":0.46100000000000002,"5min":0.33600000000000002,"15min":0.23400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.94599999999999995},"last":0.745},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59799999999999998,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.27300000000000002},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":1.4430000000000001},"last":0.67900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60799999999999998,"15min":0.56200000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.27000000000000002},"max":{"1min":0.745,"5min":0.78600000000000003,"15min":0.95899999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.20300000000000001},"max":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":1.478},"last":0.80300000000000005}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57599999999999996,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.28299999999999997},"max":{"1min":0.72699999999999998,"5min":0.83699999999999997,"15min":2.4300000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.57299999999999995},"min":{"1min":0.502,"5min":0.40600000000000003,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.89700000000000002,"15min":1.6120000000000001},"last":0.71599999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45100000000000001,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.24199999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.755},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.28000000000000003},"max":{"1min":0.70599999999999996,"5min":0.83399999999999996,"15min":0.95899999999999996},"last":0.626}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":0.45400000000000001},"min":{"1min":0.35499999999999998,"5min":0.30099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72899999999999998,"15min":0.88900000000000001},"last":0.502},{"interface":"front","average":{"1min":0.44,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":2.2400000000000002},"last":0.57699999999999996}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.77600000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.52500000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.79300000000000004,"5min":0.89700000000000002,"15min":0.92800000000000005},"last":0.55000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312012,"kb_used_data":6340,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059828,"statfs":{"total":96636764160,"available":96317263872,"internally_reserved":0,"allocated":6492160,"data_stored":3106945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.34599999999999997,"15min":0.27000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":5.0250000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.874},"min":{"1min":0.46600000000000003,"5min":0.37,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":0.85699999999999998,"15min":71.516999999999996},"last":0.45400000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.94699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.25900000000000001},"max":{"1min":0.69399999999999995,"5min":0.84199999999999997,"15min":93.381},"last":0.5},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56100000000000005,"15min":0.86399999999999999},"min":{"1min":0.39300000000000002,"5min":0.33500000000000002,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.90900000000000003,"15min":79.905000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.66400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.93899999999999995,"15min":29.305},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.318},"max":{"1min":0.66500000000000004,"5min":0.85499999999999998,"15min":9.8510000000000009},"last":0.54800000000000004}]},{"osd":3,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.69499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.255},"max":{"1min":0.75900000000000001,"5min":0.96599999999999997,"15min":34.877000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.29299999999999998},"max":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":31.657},"last":0.51000000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.51000000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.20300000000000001},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":20.212},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.442,"15min":0.46300000000000002},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.64700000000000002,"5min":0.71599999999999997,"15min":5.6909999999999998},"last":0.35999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.215},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80000000000000004},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.248},"max":{"1min":0.71499999999999997,"5min":0.81799999999999995,"15min":3.012},"last":0.58699999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47099999999999997,"15min":0.59499999999999997},"min":{"1min":0.35099999999999998,"5min":0.32500000000000001,"15min":0.24299999999999999},"max":{"1min":0.68999999999999995,"5min":0.71299999999999997,"15min":28.756},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.48899999999999999,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.32000000000000001,"15min":0.249},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":27.602},"last":0.436}]}]},{"osd":5,"up_from":13,"seq":55834575032,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":6352,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":6504448,"data_stored":3112540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.69899999999999995},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":17.704999999999998},"last":0.435},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.621,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.78100000000000003,"5min":0.872,"15min":17.465},"last":0.54900000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.22900000000000001},"max":{"1min":0.81999999999999995,"5min":0.83999999999999997,"15min":7.1710000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.42999999999999999,"5min":0.33300000000000002,"15min":0.27700000000000002},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":6.7999999999999998},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1000000000000001,"15min":8.2789999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.60299999999999998},"min":{"1min":0.51600000000000001,"5min":0.33800000000000002,"15min":0.29099999999999998},"max":{"1min":0.79300000000000004,"5min":1.133,"15min":3.0059999999999998},"last":0.61299999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.628},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.25700000000000001},"max":{"1min":0.81299999999999994,"5min":0.872,"15min":7.8090000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63600000000000001,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.38200000000000001,"15min":0.28599999999999998},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":8.6829999999999998},"last":0.72399999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46300000000000002,"15min":0.44600000000000001},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.216},"max":{"1min":0.68100000000000005,"5min":0.82099999999999995,"15min":0.98699999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.23000000000000001},"max":{"1min":0.68300000000000005,"5min":0.78000000000000003,"15min":2.0750000000000002},"last":0.63200000000000001}]},{"osd":6,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.22800000000000001},"max":{"1min":0.67000000000000004,"5min":0.81799999999999995,"15min":3.4420000000000002},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.39100000000000001,"5min":0.33300000000000002,"15min":0.25},"max":{"1min":0.68899999999999995,"5min":0.86299999999999999,"15min":3.0510000000000002},"last":0.65200000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.253},"max":{"1min":0.69699999999999995,"5min":0.79400000000000004,"15min":2.9870000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.94399999999999995,"15min":1.337},"last":0.78200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307884,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063956,"statfs":{"total":96636764160,"available":96321490944,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.86299999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.316},"max":{"1min":0.72099999999999997,"5min":0.873,"15min":0.93100000000000005},"last":0.72699999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.69699999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.32800000000000001},"max":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.379,"15min":0.223},"max":{"1min":0.73699999999999999,"5min":0.89400000000000002,"15min":0.94199999999999995},"last":0.754},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.83499999999999996}]},{"osd":3,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.47799999999999998,"5min":0.36699999999999999,"15min":0.26800000000000002},"max":{"1min":0.82499999999999996,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60399999999999998,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.36099999999999999},"max":{"1min":0.752,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.23200000000000001},"max":{"1min":0.67400000000000004,"5min":0.69699999999999995,"15min":0.81599999999999995},"last":0.47999999999999998}]},{"osd":6,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.497},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42099999999999999,"15min":0.44700000000000001},"min":{"1min":0.316,"5min":0.31,"15min":0.251},"max":{"1min":0.505,"5min":0.65700000000000003,"15min":0.69699999999999995},"last":0.59599999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.78600000000000003,"5min":1.48,"15min":1.48},"last":0.439},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.49199999999999999,"15min":0.50600000000000001},"min":{"1min":0.373,"5min":0.31900000000000001,"15min":0.28000000000000003},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.81799999999999995},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575032,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307892,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063948,"statfs":{"total":96636764160,"available":96321482752,"internally_reserved":0,"allocated":6516736,"data_stored":3118909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.371,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.52200000000000002,"5min":0.72399999999999998,"15min":29.562999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47899999999999998,"15min":0.44400000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":0.88600000000000001,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":0.29099999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.57199999999999995},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.19600000000000001},"max":{"1min":0.64400000000000002,"5min":1.6859999999999999,"15min":35.128},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.439,"15min":0.56999999999999995},"min":{"1min":0.28199999999999997,"5min":0.26600000000000001,"15min":0.24299999999999999},"max":{"1min":0.54600000000000004,"5min":1.252,"15min":29.631},"last":0.49299999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46200000000000002,"15min":0.60399999999999998},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.23000000000000001},"max":{"1min":0.622,"5min":0.90000000000000002,"15min":35.159999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.61099999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.63900000000000001,"5min":0.88600000000000001,"15min":29.603000000000002},"last":0.45900000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63,"15min":0.65900000000000003},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":0.80200000000000005,"5min":0.93899999999999995,"15min":7.3499999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64100000000000001,"15min":0.79300000000000004},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":1.0569999999999999,"15min":35.188000000000002},"last":0.69399999999999995}]},{"osd":5,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.72699999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84999999999999998,"5min":1.165,"15min":23.082999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":0.93899999999999995,"5min":1.323,"15min":7.4020000000000001},"last":0.61899999999999999}]},{"osd":6,"last update":"Sun Sep 15 02:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67300000000000004,"15min":0.78900000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.373},"max":{"1min":0.97699999999999998,"5min":1.365,"15min":35.061999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64600000000000002,"15min":0.74399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44},"max":{"1min":0.90900000000000003,"5min":1.018,"15min":23.129000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.5,"5min":0.49199999999999999,"15min":0.36699999999999999},"max":{"1min":0.88,"5min":1.1060000000000001,"15min":2.1840000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.80900000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.42099999999999999},"max":{"1min":0.81899999999999995,"5min":0.999,"15min":35.100999999999999},"last":0.80100000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575031,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312044,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059796,"statfs":{"total":96636764160,"available":96317231104,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.253,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":0.55200000000000005,"5min":0.65000000000000002,"15min":1.2370000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":0.57199999999999995,"5min":0.747,"15min":1.1799999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.27100000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59099999999999997,"5min":0.70999999999999996,"15min":1.319},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.73799999999999999,"15min":1.347},"last":0.52700000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.34899999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.55900000000000005,"5min":0.66700000000000004,"15min":1.411},"last":0.502},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.34599999999999997,"5min":0.29199999999999998,"15min":0.252},"max":{"1min":0.622,"5min":0.67500000000000004,"15min":2.9340000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.73099999999999998,"5min":0.95499999999999996,"15min":1.298},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.61699999999999999},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.875,"15min":1.1319999999999999},"last":0.625}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.315},"max":{"1min":0.81999999999999995,"5min":1,"15min":2.9079999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.49199999999999999,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.94999999999999996,"15min":1.367},"last":0.79900000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.008,"15min":1.2050000000000001},"last":0.753},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.96999999999999997,"15min":2.9649999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.50800000000000001,"5min":0.43099999999999999,"15min":0.31900000000000001},"max":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":1.3240000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.44600000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":0.82099999999999995,"5min":0.95199999999999996,"15min":1.167},"last":0.81799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303724,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068116,"statfs":{"total":96636764160,"available":96325750784,"internally_reserved":0,"allocated":6516736,"data_stored":3118923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.38900000000000001,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.17899999999999999},"max":{"1min":0.70399999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51800000000000002,"5min":0.65100000000000002,"15min":0.93100000000000005},"last":0.39500000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56000000000000005,"5min":1.405,"15min":1.405},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.441},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.219},"max":{"1min":0.64700000000000002,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.30099999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.249},"max":{"1min":0.81000000000000005,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.67100000000000004,"5min":2.8490000000000002,"15min":2.8490000000000002},"last":0.38}]},{"osd":4,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63200000000000001,"15min":0.59999999999999998},"min":{"1min":0.48899999999999999,"5min":0.379,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":1.581,"15min":1.581},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63800000000000001,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.34899999999999998,"15min":0.29299999999999998},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.95499999999999996},"last":0.65100000000000002}]},{"osd":5,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.64200000000000002},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.33300000000000002},"max":{"1min":0.90200000000000002,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.65800000000000003},"min":{"1min":0.56200000000000006,"5min":0.433,"15min":0.371},"max":{"1min":1.004,"5min":1.532,"15min":1.532},"last":0.67800000000000005}]},{"osd":6,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.40400000000000003,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.70299999999999996,"15min":0.66300000000000003},"min":{"1min":0.60899999999999999,"5min":0.46800000000000003,"15min":0.33000000000000002},"max":{"1min":1.0509999999999999,"5min":1.498,"15min":1.498},"last":0.63}]},{"osd":7,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.53500000000000003,"5min":0.35999999999999999,"15min":0.35699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.32500000000000001},"max":{"1min":0.97399999999999998,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.59599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307920,"kb_used_data":6400,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063920,"statfs":{"total":96636764160,"available":96321454080,"internally_reserved":0,"allocated":6553600,"data_stored":3641449,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.503},"min":{"1min":0.251,"5min":0.24199999999999999,"15min":0.182},"max":{"1min":0.64800000000000002,"5min":0.68500000000000005,"15min":14.630000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46999999999999997,"15min":0.45100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.222},"max":{"1min":0.69299999999999995,"5min":0.85399999999999998,"15min":2.3570000000000002},"last":0.60699999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44500000000000001,"15min":0.47499999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.20200000000000001},"max":{"1min":0.67000000000000004,"5min":0.93899999999999995,"15min":14.679},"last":0.503},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":0.68600000000000005,"15min":1.806},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.21099999999999999},"max":{"1min":0.504,"5min":0.73799999999999999,"15min":2.835},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.48899999999999999,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.26300000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":14.558999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64900000000000002,"15min":0.60499999999999998},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.253},"max":{"1min":0.85599999999999998,"5min":0.98599999999999999,"15min":1.5549999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65300000000000002,"15min":0.61699999999999999},"min":{"1min":0.38900000000000001,"5min":0.33200000000000002,"15min":0.30199999999999999},"max":{"1min":0.79000000000000004,"5min":0.94699999999999995,"15min":1.3120000000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63400000000000001,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.32800000000000001},"max":{"1min":0.878,"5min":0.998,"15min":1.1579999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.34300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":7.2969999999999997},"last":0.67500000000000004}]},{"osd":6,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68200000000000005,"15min":0.70599999999999996},"min":{"1min":0.55700000000000005,"5min":0.48599999999999999,"15min":0.35199999999999998},"max":{"1min":0.97499999999999998,"5min":1.0129999999999999,"15min":14.6},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69599999999999995,"15min":0.65400000000000003},"min":{"1min":0.53200000000000003,"5min":0.40999999999999998,"15min":0.35399999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.79600000000000004}]},{"osd":7,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67900000000000005,"15min":0.64000000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.36699999999999999},"max":{"1min":0.95399999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.34300000000000003},"max":{"1min":0.97899999999999998,"5min":1.0029999999999999,"15min":1.093},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:58:25.273 INFO:tasks.ceph:pgid 12.10 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.273 INFO:tasks.ceph:pgid 12.11 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.12 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.13 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.14 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.15 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.16 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.17 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.18 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.19 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.274 INFO:tasks.ceph:pgid 12.1a last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.275 INFO:tasks.ceph:pgid 12.1b last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.275 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.275 INFO:tasks.ceph:pgid 12.9 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.275 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.276 INFO:tasks.ceph:pgid 12.8 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.276 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.276 INFO:tasks.ceph:pgid 12.b last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.276 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.276 INFO:tasks.ceph:pgid 12.a last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 12.c last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 12.e last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T02:43:20.381248+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.277 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T02:43:19.374225+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=43, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.278 INFO:tasks.ceph:pgid 12.d last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.278 INFO:tasks.ceph:pgid 12.7 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.6 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.5 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.4 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.3 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.2 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.1 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.0 last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.279 INFO:tasks.ceph:pgid 12.1f last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.280 INFO:tasks.ceph:pgid 12.1e last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.280 INFO:tasks.ceph:pgid 12.1d last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.280 INFO:tasks.ceph:pgid 12.1c last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:25.280 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T02:58:45.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:58:45.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.390+0000 7f3b7195d640 10 monclient: get_monmap_and_config 2024-09-15T02:58:45.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.390+0000 7f3b7195d640 10 monclient: build_initial_monmap 2024-09-15T02:58:45.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.390+0000 7f3b7195d640 10 monclient: monmap: 2024-09-15T02:58:45.392 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:45.392 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:45.392 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:58:45.392217+0000 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:58:45.392217+0000 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:58:45.393 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 1 Processor -- start 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 1 -- start start 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient: init 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3b6c0b0aa0 0x7f3b6c13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b6c0b0aa0 addr v2:172.21.15.104:3300/0 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3b6c06e2f0 0x7f3b6c06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b6c06e2f0 addr v2:172.21.15.53:3300/0 2024-09-15T02:58:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b6c06ec60 addr v1:172.21.15.104:6789/0 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c13e8a0 con 0x7f3b6c06ec60 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c0b0e70 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c06d660 con 0x7f3b6c0b0aa0 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b7195d640 10 monclient(hunting): authenticate will time out at 1661.962810s 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b6a7fc640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3b6c06e2f0 0x7f3b6c06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b6a7fc640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3b6c06e2f0 0x7f3b6c06e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58616/0 (socket says 172.21.15.53:58616) 2024-09-15T02:58:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b6a7fc640 1 -- 172.21.15.53:0/3609643310 learned_addr learned my addr 172.21.15.53:0/3609643310 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b6a7fc640 10 monclient(hunting): get_auth_request con 0x7f3b6c06e2f0 auth_method 0 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.391+0000 7f3b6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6affd640 1 --2- 172.21.15.53:0/3609643310 >> v2:172.21.15.104:3300/0 conn(0x7f3b6c0b0aa0 0x7f3b6c13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6affd640 10 monclient(hunting): get_auth_request con 0x7f3b6c0b0aa0 auth_method 0 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient(hunting): handle_auth_done global_id 26355 payload 274 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 1 -- 172.21.15.53:0/3609643310 >> v2:172.21.15.104:3300/0 conn(0x7f3b6c0b0aa0 msgr2=0x7f3b6c13dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 1 --2- 172.21.15.53:0/3609643310 >> v2:172.21.15.104:3300/0 conn(0x7f3b6c0b0aa0 0x7f3b6c13dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 1 -- 172.21.15.53:0/3609643310 >> v1:172.21.15.104:6789/0 conn(0x7f3b6c06ec60 legacy=0x7f3b6c13a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 1 monclient: found mon.noname-a 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6a7fc640 1 -- 172.21.15.53:0/3609643310 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c06d940 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.392+0000 7f3b6affd640 1 --2- 172.21.15.53:0/3609643310 >> v2:172.21.15.104:3300/0 conn(0x7f3b6c0b0aa0 0x7f3b6c13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b6a7fc640 1 --2- 172.21.15.53:0/3609643310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c06e720 secure :-1 s=READY pgs=5505 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c068880 tx=0x7f3b600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1c64cd7dcd089c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 1 -- 172.21.15.53:0/3609643310 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6002ae80 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: dump: 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: _finish_auth 0 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 1 -- 172.21.15.53:0/3609643310 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b60032440 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 5 monclient: authenticate success, global_id 26355 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 10 monclient: get_monmap_and_config success 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 1 -- 172.21.15.53:0/3609643310 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60032800 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 4 set_mon_vals no callback set 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b69ffb640 10 monclient: dump: 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:45.403 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:45.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 10 monclient: shutdown 2024-09-15T02:58:45.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:45.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 1 -- 172.21.15.53:0/3609643310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 msgr2=0x7f3b6c06e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:45.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 1 --2- 172.21.15.53:0/3609643310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c06e720 secure :-1 s=READY pgs=5505 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c068880 tx=0x7f3b600091d0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 1 -- 172.21.15.53:0/3609643310 shutdown_connections 2024-09-15T02:58:45.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 1 --2- 172.21.15.53:0/3609643310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c06e720 unknown :-1 s=CLOSED pgs=5505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 1 --2- 172.21.15.53:0/3609643310 >> v2:172.21.15.104:3300/0 conn(0x7f3b6c0b0aa0 0x7f3b6c13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 1 -- 172.21.15.53:0/3609643310 >> 172.21.15.53:0/3609643310 conn(0x7f3b6c067b00 msgr2=0x7f3b6c0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.393+0000 7f3b7195d640 1 -- 172.21.15.53:0/3609643310 shutdown_connections 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.394+0000 7f3b7195d640 1 -- 172.21.15.53:0/3609643310 wait complete. 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.394+0000 7f3b7195d640 10 monclient: build_initial_monmap 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.394+0000 7f3b7195d640 10 monclient: monmap: 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:58:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 librados: starting msgr at 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 librados: starting objecter 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 Processor -- start 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 -- start start 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 librados: setting wanted keys 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 librados: calling monclient init 2024-09-15T02:58:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient: init 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient: _reopen_session rank -1 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c0aa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b6c06e2f0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b6c06ec60 0x7f3b6c14d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b6c06ec60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b6c0b0aa0 0x7f3b6c14de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:45.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b6c0b0aa0 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0a8e20 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0a9f10 con 0x7f3b6c06ec60 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0a9910 con 0x7f3b6c0b0aa0 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): _renew_subs 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b7195d640 10 monclient(hunting): authenticate will time out at 1661.966920s 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b6affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c0aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.395+0000 7f3b6affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c0aa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58626/0 (socket says 172.21.15.53:58626) 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 -- 172.21.15.53:0/1563506117 learned_addr learned my addr 172.21.15.53:0/1563506117 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): get_auth_request con 0x7f3b6c06e2f0 auth_method 0 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:45.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b6c0b0aa0 0x7f3b6c14de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b6c06ec60 0x7f3b6c14d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 10 monclient(hunting): get_auth_request con 0x7f3b6c06ec60 auth_method 0 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 10 monclient(hunting): get_auth_request con 0x7f3b6c0b0aa0 auth_method 0 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:58:45.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient(hunting): handle_auth_done global_id 26358 payload 824 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 -- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b6c0b0aa0 msgr2=0x7f3b6c14de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b6c0b0aa0 0x7f3b6c14de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 -- 172.21.15.53:0/1563506117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b6c06ec60 msgr2=0x7f3b6c14d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b6c06ec60 0x7f3b6c14d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient: _finish_hunting 0 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 monclient: found mon.noname-a 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 -- 172.21.15.53:0/1563506117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b60009450 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b6affd640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c0aa480 secure :-1 s=READY pgs=5506 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00fe20 tx=0x7f3b5c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 client.?.objecter ms_handle_connect 0x7f3b6c06e2f0 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 1 -- 172.21.15.53:0/1563506117 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c00de10 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: dump: 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:45.416 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: _finish_auth 0 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 1 -- 172.21.15.53:0/1563506117 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c009d40 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:58:45.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 1 -- 172.21.15.53:0/1563506117 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0116a0 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 5 monclient: authenticate success, global_id 26358 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b52ffd640 10 monclient: dump: 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:58:45.418 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 10 monclient: _renew_subs 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c0a9ad0 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b697fa640 4 set_mon_vals no callback set 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 10 client.26358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 10 monclient: _renew_subs 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:45.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c14e5b0 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.396+0000 7f3b7195d640 1 librados: init done 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.397+0000 7f3b7195d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.397+0000 7f3b7195d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.397+0000 7f3b7195d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.397+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c0aa0d0 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.398+0000 7f3b52ffd640 1 -- 172.21.15.53:0/1563506117 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f3b5c011a50 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.398+0000 7f3b52ffd640 1 --2- 172.21.15.53:0/1563506117 >> v2:172.21.15.104:6816/32804 conn(0x7f3b5806e800 0x7f3b58070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.398+0000 7f3b52ffd640 1 -- 172.21.15.53:0/1563506117 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 502..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f3b5c08e2d0 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.398+0000 7f3b52ffd640 10 client.26358.objecter ms_dispatch 0x7f3b6c000b90 osd_map(702..702 src has 502..702) v4 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.398+0000 7f3b52ffd640 3 client.26358.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.398+0000 7f3b52ffd640 3 client.26358.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.399+0000 7f3b52ffd640 20 client.26358.objecter dump_active .. 0 homeless 2024-09-15T02:58:45.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.399+0000 7f3b6a7fc640 1 --2- 172.21.15.53:0/1563506117 >> v2:172.21.15.104:6816/32804 conn(0x7f3b5806e800 0x7f3b58070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:58:45.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.399+0000 7f3b6a7fc640 10 monclient: get_auth_request con 0x7f3b5806e800 auth_method 0 2024-09-15T02:58:45.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.399+0000 7f3b52ffd640 10 client.26358.objecter ms_handle_connect 0x7f3b5806e800 2024-09-15T02:58:45.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.399+0000 7f3b6a7fc640 1 --2- 172.21.15.53:0/1563506117 >> v2:172.21.15.104:6816/32804 conn(0x7f3b5806e800 0x7f3b58070cb0 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7f3b60034000 tx=0x7f3b60032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:58:45.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.403+0000 7f3b52ffd640 1 -- 172.21.15.53:0/1563506117 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3b5c058570 con 0x7f3b6c06e2f0 2024-09-15T02:58:45.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.403+0000 7f3b52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:58:45.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.403+0000 7f3b52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:58:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.562+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b6c0646e0 con 0x7f3b5806e800 2024-09-15T02:58:45.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.567+0000 7f3b52ffd640 1 -- 172.21.15.53:0/1563506117 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+154398 (secure 0 0 0) 0x7f3b6c0646e0 con 0x7f3b5806e800 2024-09-15T02:58:45.572 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:58:45.573 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:58:45.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 10 librados: watch_flush enter 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 10 librados: watch_flush exit 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 20 client.26358.objecter shutdown clearing up homeless session... 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 10 client.26358.objecter successfully canceled tick 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 >> v2:172.21.15.104:6816/32804 conn(0x7f3b5806e800 msgr2=0x7f3b58070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 --2- 172.21.15.53:0/1563506117 >> v2:172.21.15.104:6816/32804 conn(0x7f3b5806e800 0x7f3b58070cb0 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7f3b60034000 tx=0x7f3b60032000 comp rx=0 tx=0).stop 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 10 monclient: shutdown 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 msgr2=0x7f3b6c0aa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c0aa480 secure :-1 s=READY pgs=5506 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00fe20 tx=0x7f3b5c008040 comp rx=0 tx=0).stop 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 shutdown_connections 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b6c0b0aa0 0x7f3b6c14de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 --2- 172.21.15.53:0/1563506117 >> v2:172.21.15.104:6816/32804 conn(0x7f3b5806e800 0x7f3b58070cb0 unknown :-1 s=CLOSED pgs=3709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b6c06ec60 0x7f3b6c14d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 --2- 172.21.15.53:0/1563506117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b6c06e2f0 0x7f3b6c0aa480 unknown :-1 s=CLOSED pgs=5506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:58:45.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 >> 172.21.15.53:0/1563506117 conn(0x7f3b6c067b00 msgr2=0x7f3b6c1379c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:58:45.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 shutdown_connections 2024-09-15T02:58:45.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.569+0000 7f3b7195d640 1 -- 172.21.15.53:0/1563506117 wait complete. 2024-09-15T02:58:45.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:58:45.570+0000 7f3b7195d640 1 librados: shutdown 2024-09-15T02:58:45.582 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":1158,"stamp":"2024-09-15T02:58:45.545936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2471416,"kb_used_data":51000,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752503304,"statfs":{"total":773094113280,"available":770563383296,"internally_reserved":0,"allocated":52224000,"data_stored":26009136,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002858"},"pg_stats":[{"pgid":"12.10","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.309703+0000","last_change":"2024-09-15T02:58:29.309675+0000","last_active":"2024-09-15T02:58:29.309703+0000","last_peered":"2024-09-15T02:58:29.309703+0000","last_clean":"2024-09-15T02:58:29.309703+0000","last_became_active":"2024-09-15T02:44:21.589224+0000","last_became_peered":"2024-09-15T02:44:21.589224+0000","last_unstale":"2024-09-15T02:58:29.309703+0000","last_undegraded":"2024-09-15T02:58:29.309703+0000","last_fullsized":"2024-09-15T02:58:29.309703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.309591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.309591+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.309591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:04:48.245075+0000","scrub_duration":0.001767556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":656,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.249325+0000","last_change":"2024-09-15T02:58:33.249295+0000","last_active":"2024-09-15T02:58:33.249325+0000","last_peered":"2024-09-15T02:58:33.249325+0000","last_clean":"2024-09-15T02:58:33.249325+0000","last_became_active":"2024-09-15T02:44:21.472608+0000","last_became_peered":"2024-09-15T02:44:21.472608+0000","last_unstale":"2024-09-15T02:58:33.249325+0000","last_undegraded":"2024-09-15T02:58:33.249325+0000","last_fullsized":"2024-09-15T02:58:33.249325+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.249211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.249211+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.249211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:54:05.493624+0000","scrub_duration":0.0013138430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:27.193298+0000","last_change":"2024-09-15T02:58:27.193272+0000","last_active":"2024-09-15T02:58:27.193298+0000","last_peered":"2024-09-15T02:58:27.193298+0000","last_clean":"2024-09-15T02:58:27.193298+0000","last_became_active":"2024-09-15T02:44:21.604946+0000","last_became_peered":"2024-09-15T02:44:21.604946+0000","last_unstale":"2024-09-15T02:58:27.193298+0000","last_undegraded":"2024-09-15T02:58:27.193298+0000","last_fullsized":"2024-09-15T02:58:27.193298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:27.193179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:27.193179+0000","last_clean_scrub_stamp":"2024-09-15T02:58:27.193179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:59:07.330194+0000","scrub_duration":0.0015378810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:40.523869+0000","last_change":"2024-09-15T02:58:40.523839+0000","last_active":"2024-09-15T02:58:40.523869+0000","last_peered":"2024-09-15T02:58:40.523869+0000","last_clean":"2024-09-15T02:58:40.523869+0000","last_became_active":"2024-09-15T02:44:21.621686+0000","last_became_peered":"2024-09-15T02:44:21.621686+0000","last_unstale":"2024-09-15T02:58:40.523869+0000","last_undegraded":"2024-09-15T02:58:40.523869+0000","last_fullsized":"2024-09-15T02:58:40.523869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:40.523733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:40.523733+0000","last_clean_scrub_stamp":"2024-09-15T02:58:40.523733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:38:26.536590+0000","scrub_duration":0.001444007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":656,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.285854+0000","last_change":"2024-09-15T02:58:29.285826+0000","last_active":"2024-09-15T02:58:29.285854+0000","last_peered":"2024-09-15T02:58:29.285854+0000","last_clean":"2024-09-15T02:58:29.285854+0000","last_became_active":"2024-09-15T02:44:21.558951+0000","last_became_peered":"2024-09-15T02:44:21.558951+0000","last_unstale":"2024-09-15T02:58:29.285854+0000","last_undegraded":"2024-09-15T02:58:29.285854+0000","last_fullsized":"2024-09-15T02:58:29.285854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.285737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.285737+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.285737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:11:02.269546+0000","scrub_duration":0.0020859030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:37.405564+0000","last_change":"2024-09-15T02:58:37.405540+0000","last_active":"2024-09-15T02:58:37.405564+0000","last_peered":"2024-09-15T02:58:37.405564+0000","last_clean":"2024-09-15T02:58:37.405564+0000","last_became_active":"2024-09-15T02:44:21.592665+0000","last_became_peered":"2024-09-15T02:44:21.592665+0000","last_unstale":"2024-09-15T02:58:37.405564+0000","last_undegraded":"2024-09-15T02:58:37.405564+0000","last_fullsized":"2024-09-15T02:58:37.405564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:37.405455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:37.405455+0000","last_clean_scrub_stamp":"2024-09-15T02:58:37.405455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:23:31.478408+0000","scrub_duration":0.0017888019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:39.369461+0000","last_change":"2024-09-15T02:58:39.369433+0000","last_active":"2024-09-15T02:58:39.369461+0000","last_peered":"2024-09-15T02:58:39.369461+0000","last_clean":"2024-09-15T02:58:39.369461+0000","last_became_active":"2024-09-15T02:44:21.537279+0000","last_became_peered":"2024-09-15T02:44:21.537279+0000","last_unstale":"2024-09-15T02:58:39.369461+0000","last_undegraded":"2024-09-15T02:58:39.369461+0000","last_fullsized":"2024-09-15T02:58:39.369461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:39.369332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:39.369332+0000","last_clean_scrub_stamp":"2024-09-15T02:58:39.369332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:59.952893+0000","scrub_duration":0.0013346740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:44.409095+0000","last_change":"2024-09-15T02:58:44.409072+0000","last_active":"2024-09-15T02:58:44.409095+0000","last_peered":"2024-09-15T02:58:44.409095+0000","last_clean":"2024-09-15T02:58:44.409095+0000","last_became_active":"2024-09-15T02:44:21.612841+0000","last_became_peered":"2024-09-15T02:44:21.612841+0000","last_unstale":"2024-09-15T02:58:44.409095+0000","last_undegraded":"2024-09-15T02:58:44.409095+0000","last_fullsized":"2024-09-15T02:58:44.409095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:44.408977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:44.408977+0000","last_clean_scrub_stamp":"2024-09-15T02:58:44.408977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:29:04.777876+0000","scrub_duration":0.0018179050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:41.561370+0000","last_change":"2024-09-15T02:58:41.561341+0000","last_active":"2024-09-15T02:58:41.561370+0000","last_peered":"2024-09-15T02:58:41.561370+0000","last_clean":"2024-09-15T02:58:41.561370+0000","last_became_active":"2024-09-15T02:44:21.522860+0000","last_became_peered":"2024-09-15T02:44:21.522860+0000","last_unstale":"2024-09-15T02:58:41.561370+0000","last_undegraded":"2024-09-15T02:58:41.561370+0000","last_fullsized":"2024-09-15T02:58:41.561370+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:41.561248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:41.561248+0000","last_clean_scrub_stamp":"2024-09-15T02:58:41.561248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:56:26.494114+0000","scrub_duration":0.0018629569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:24.251933+0000","last_change":"2024-09-15T02:58:24.251914+0000","last_active":"2024-09-15T02:58:24.251933+0000","last_peered":"2024-09-15T02:58:24.251933+0000","last_clean":"2024-09-15T02:58:24.251933+0000","last_became_active":"2024-09-15T02:44:21.535334+0000","last_became_peered":"2024-09-15T02:44:21.535334+0000","last_unstale":"2024-09-15T02:58:24.251933+0000","last_undegraded":"2024-09-15T02:58:24.251933+0000","last_fullsized":"2024-09-15T02:58:24.251933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:24.251846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:24.251846+0000","last_clean_scrub_stamp":"2024-09-15T02:58:24.251846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:16:17.676483+0000","scrub_duration":0.0014797040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:32.440561+0000","last_change":"2024-09-15T02:58:32.440535+0000","last_active":"2024-09-15T02:58:32.440561+0000","last_peered":"2024-09-15T02:58:32.440561+0000","last_clean":"2024-09-15T02:58:32.440561+0000","last_became_active":"2024-09-15T02:44:21.614186+0000","last_became_peered":"2024-09-15T02:44:21.614186+0000","last_unstale":"2024-09-15T02:58:32.440561+0000","last_undegraded":"2024-09-15T02:58:32.440561+0000","last_fullsized":"2024-09-15T02:58:32.440561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:32.440444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:32.440444+0000","last_clean_scrub_stamp":"2024-09-15T02:58:32.440444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:23:30.652467+0000","scrub_duration":0.001723069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.487552+0000","last_change":"2024-09-15T02:58:33.487527+0000","last_active":"2024-09-15T02:58:33.487552+0000","last_peered":"2024-09-15T02:58:33.487552+0000","last_clean":"2024-09-15T02:58:33.487552+0000","last_became_active":"2024-09-15T02:44:21.527331+0000","last_became_peered":"2024-09-15T02:44:21.527331+0000","last_unstale":"2024-09-15T02:58:33.487552+0000","last_undegraded":"2024-09-15T02:58:33.487552+0000","last_fullsized":"2024-09-15T02:58:33.487552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.487436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.487436+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.487436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:55:46.304879+0000","scrub_duration":0.0015127599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":665,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:26.399656+0000","last_change":"2024-09-15T02:58:26.399610+0000","last_active":"2024-09-15T02:58:26.399656+0000","last_peered":"2024-09-15T02:58:26.399656+0000","last_clean":"2024-09-15T02:58:26.399656+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:58:26.399656+0000","last_undegraded":"2024-09-15T02:58:26.399656+0000","last_fullsized":"2024-09-15T02:58:26.399656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:26.399523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:26.399523+0000","last_clean_scrub_stamp":"2024-09-15T02:58:26.399523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:56:49.179797+0000","scrub_duration":0.001359159,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":663,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:32.753013+0000","last_change":"2024-09-15T02:58:32.752984+0000","last_active":"2024-09-15T02:58:32.753013+0000","last_peered":"2024-09-15T02:58:32.753013+0000","last_clean":"2024-09-15T02:58:32.753013+0000","last_became_active":"2024-09-15T02:44:21.530274+0000","last_became_peered":"2024-09-15T02:44:21.530274+0000","last_unstale":"2024-09-15T02:58:32.753013+0000","last_undegraded":"2024-09-15T02:58:32.753013+0000","last_fullsized":"2024-09-15T02:58:32.753013+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:32.752892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:32.752892+0000","last_clean_scrub_stamp":"2024-09-15T02:58:32.752892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:31:00.482455+0000","scrub_duration":0.0017938310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:28.453457+0000","last_change":"2024-09-15T02:58:28.453430+0000","last_active":"2024-09-15T02:58:28.453457+0000","last_peered":"2024-09-15T02:58:28.453457+0000","last_clean":"2024-09-15T02:58:28.453457+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:58:28.453457+0000","last_undegraded":"2024-09-15T02:58:28.453457+0000","last_fullsized":"2024-09-15T02:58:28.453457+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:28.453331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:28.453331+0000","last_clean_scrub_stamp":"2024-09-15T02:58:28.453331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:46:28.458820+0000","scrub_duration":0.0017649370000000001,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.260014+0000","last_change":"2024-09-15T02:58:35.259984+0000","last_active":"2024-09-15T02:58:35.260014+0000","last_peered":"2024-09-15T02:58:35.260014+0000","last_clean":"2024-09-15T02:58:35.260014+0000","last_became_active":"2024-09-15T02:44:21.579154+0000","last_became_peered":"2024-09-15T02:44:21.579154+0000","last_unstale":"2024-09-15T02:58:35.260014+0000","last_undegraded":"2024-09-15T02:58:35.260014+0000","last_fullsized":"2024-09-15T02:58:35.260014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.259895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.259895+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.259895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:40:55.314158+0000","scrub_duration":0.0015412959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":662,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.440059+0000","last_change":"2024-09-15T02:58:33.440032+0000","last_active":"2024-09-15T02:58:33.440059+0000","last_peered":"2024-09-15T02:58:33.440059+0000","last_clean":"2024-09-15T02:58:33.440059+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:58:33.440059+0000","last_undegraded":"2024-09-15T02:58:33.440059+0000","last_fullsized":"2024-09-15T02:58:33.440059+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.439945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.439945+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.439945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:32:10.258043+0000","scrub_duration":0.002092979,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.150970+0000","last_change":"2024-09-15T02:58:29.150934+0000","last_active":"2024-09-15T02:58:29.150970+0000","last_peered":"2024-09-15T02:58:29.150970+0000","last_clean":"2024-09-15T02:58:29.150970+0000","last_became_active":"2024-09-15T02:44:21.533501+0000","last_became_peered":"2024-09-15T02:44:21.533501+0000","last_unstale":"2024-09-15T02:58:29.150970+0000","last_undegraded":"2024-09-15T02:58:29.150970+0000","last_fullsized":"2024-09-15T02:58:29.150970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.150843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.150843+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.150843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:01:50.177046+0000","scrub_duration":0.0014478469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:30.510900+0000","last_change":"2024-09-15T02:58:30.510856+0000","last_active":"2024-09-15T02:58:30.510900+0000","last_peered":"2024-09-15T02:58:30.510900+0000","last_clean":"2024-09-15T02:58:30.510900+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:58:30.510900+0000","last_undegraded":"2024-09-15T02:58:30.510900+0000","last_fullsized":"2024-09-15T02:58:30.510900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:30.510759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:30.510759+0000","last_clean_scrub_stamp":"2024-09-15T02:58:30.510759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:02:35.096214+0000","scrub_duration":0.001387597,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":663,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.720687+0000","last_change":"2024-09-15T02:58:35.720631+0000","last_active":"2024-09-15T02:58:35.720687+0000","last_peered":"2024-09-15T02:58:35.720687+0000","last_clean":"2024-09-15T02:58:35.720687+0000","last_became_active":"2024-09-15T02:44:21.560425+0000","last_became_peered":"2024-09-15T02:44:21.560425+0000","last_unstale":"2024-09-15T02:58:35.720687+0000","last_undegraded":"2024-09-15T02:58:35.720687+0000","last_fullsized":"2024-09-15T02:58:35.720687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.720534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.720534+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.720534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:21:01.695265+0000","scrub_duration":0.0015548070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"702'3","reported_seq":670,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:24.219147+0000","last_change":"2024-09-15T02:58:24.219122+0000","last_active":"2024-09-15T02:58:24.219147+0000","last_peered":"2024-09-15T02:58:24.219147+0000","last_clean":"2024-09-15T02:58:24.219147+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:58:24.219147+0000","last_undegraded":"2024-09-15T02:58:24.219147+0000","last_fullsized":"2024-09-15T02:58:24.219147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'3","last_scrub_stamp":"2024-09-15T02:58:24.219040+0000","last_deep_scrub":"702'3","last_deep_scrub_stamp":"2024-09-15T02:58:24.219040+0000","last_clean_scrub_stamp":"2024-09-15T02:58:24.219040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:27:46.775672+0000","scrub_duration":0.0028255989999999998,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.356868+0000","last_change":"2024-09-15T02:58:33.356818+0000","last_active":"2024-09-15T02:58:33.356868+0000","last_peered":"2024-09-15T02:58:33.356868+0000","last_clean":"2024-09-15T02:58:33.356868+0000","last_became_active":"2024-09-15T02:44:21.500646+0000","last_became_peered":"2024-09-15T02:44:21.500646+0000","last_unstale":"2024-09-15T02:58:33.356868+0000","last_undegraded":"2024-09-15T02:58:33.356868+0000","last_fullsized":"2024-09-15T02:58:33.356868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.356708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.356708+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.356708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:52:18.376144+0000","scrub_duration":0.00155893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":674,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:30.745359+0000","last_change":"2024-09-15T02:58:30.745328+0000","last_active":"2024-09-15T02:58:30.745359+0000","last_peered":"2024-09-15T02:58:30.745359+0000","last_clean":"2024-09-15T02:58:30.745359+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:58:30.745359+0000","last_undegraded":"2024-09-15T02:58:30.745359+0000","last_fullsized":"2024-09-15T02:58:30.745359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:30.745227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:30.745227+0000","last_clean_scrub_stamp":"2024-09-15T02:58:30.745227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:25:44.715970+0000","scrub_duration":0.0014903729999999999,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:34.449874+0000","last_change":"2024-09-15T02:58:34.449844+0000","last_active":"2024-09-15T02:58:34.449874+0000","last_peered":"2024-09-15T02:58:34.449874+0000","last_clean":"2024-09-15T02:58:34.449874+0000","last_became_active":"2024-09-15T02:44:21.504665+0000","last_became_peered":"2024-09-15T02:44:21.504665+0000","last_unstale":"2024-09-15T02:58:34.449874+0000","last_undegraded":"2024-09-15T02:58:34.449874+0000","last_fullsized":"2024-09-15T02:58:34.449874+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:34.449751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:34.449751+0000","last_clean_scrub_stamp":"2024-09-15T02:58:34.449751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:29:27.629576+0000","scrub_duration":0.0015521109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":661,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.525274+0000","last_change":"2024-09-15T02:58:29.525246+0000","last_active":"2024-09-15T02:58:29.525274+0000","last_peered":"2024-09-15T02:58:29.525274+0000","last_clean":"2024-09-15T02:58:29.525274+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:58:29.525274+0000","last_undegraded":"2024-09-15T02:58:29.525274+0000","last_fullsized":"2024-09-15T02:58:29.525274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.525157+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.525157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:31:17.681291+0000","scrub_duration":0.001556654,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:32.372925+0000","last_change":"2024-09-15T02:58:32.372896+0000","last_active":"2024-09-15T02:58:32.372925+0000","last_peered":"2024-09-15T02:58:32.372925+0000","last_clean":"2024-09-15T02:58:32.372925+0000","last_became_active":"2024-09-15T02:44:21.644939+0000","last_became_peered":"2024-09-15T02:44:21.644939+0000","last_unstale":"2024-09-15T02:58:32.372925+0000","last_undegraded":"2024-09-15T02:58:32.372925+0000","last_fullsized":"2024-09-15T02:58:32.372925+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:32.372804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:32.372804+0000","last_clean_scrub_stamp":"2024-09-15T02:58:32.372804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:46:40.555477+0000","scrub_duration":0.001735297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"702'2","reported_seq":669,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:27.281177+0000","last_change":"2024-09-15T02:58:27.281138+0000","last_active":"2024-09-15T02:58:27.281177+0000","last_peered":"2024-09-15T02:58:27.281177+0000","last_clean":"2024-09-15T02:58:27.281177+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:58:27.281177+0000","last_undegraded":"2024-09-15T02:58:27.281177+0000","last_fullsized":"2024-09-15T02:58:27.281177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'2","last_scrub_stamp":"2024-09-15T02:58:27.281056+0000","last_deep_scrub":"702'2","last_deep_scrub_stamp":"2024-09-15T02:58:27.281056+0000","last_clean_scrub_stamp":"2024-09-15T02:58:27.281056+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:54:37.158852+0000","scrub_duration":0.0033533730000000002,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"702'74","reported_seq":749,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:25.589886+0000","last_change":"2024-09-15T02:58:25.589861+0000","last_active":"2024-09-15T02:58:25.589886+0000","last_peered":"2024-09-15T02:58:25.589886+0000","last_clean":"2024-09-15T02:58:25.589886+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:58:25.589886+0000","last_undegraded":"2024-09-15T02:58:25.589886+0000","last_fullsized":"2024-09-15T02:58:25.589886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'74","last_scrub_stamp":"2024-09-15T02:58:25.589784+0000","last_deep_scrub":"702'74","last_deep_scrub_stamp":"2024-09-15T02:58:25.589784+0000","last_clean_scrub_stamp":"2024-09-15T02:58:25.589784+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:33:24.442211+0000","scrub_duration":0.0044050640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":651,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:38.345943+0000","last_change":"2024-09-15T02:58:38.345914+0000","last_active":"2024-09-15T02:58:38.345943+0000","last_peered":"2024-09-15T02:58:38.345943+0000","last_clean":"2024-09-15T02:58:38.345943+0000","last_became_active":"2024-09-15T02:44:21.629239+0000","last_became_peered":"2024-09-15T02:44:21.629239+0000","last_unstale":"2024-09-15T02:58:38.345943+0000","last_undegraded":"2024-09-15T02:58:38.345943+0000","last_fullsized":"2024-09-15T02:58:38.345943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:38.345813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:38.345813+0000","last_clean_scrub_stamp":"2024-09-15T02:58:38.345813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:51:52.034818+0000","scrub_duration":0.0018938099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:36.447577+0000","last_change":"2024-09-15T02:58:36.447552+0000","last_active":"2024-09-15T02:58:36.447577+0000","last_peered":"2024-09-15T02:58:36.447577+0000","last_clean":"2024-09-15T02:58:36.447577+0000","last_became_active":"2024-09-15T02:44:21.571415+0000","last_became_peered":"2024-09-15T02:44:21.571415+0000","last_unstale":"2024-09-15T02:58:36.447577+0000","last_undegraded":"2024-09-15T02:58:36.447577+0000","last_fullsized":"2024-09-15T02:58:36.447577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:36.447460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:36.447460+0000","last_clean_scrub_stamp":"2024-09-15T02:58:36.447460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:34.347332+0000","scrub_duration":0.001684409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.494257+0000","last_change":"2024-09-15T02:58:35.494232+0000","last_active":"2024-09-15T02:58:35.494257+0000","last_peered":"2024-09-15T02:58:35.494257+0000","last_clean":"2024-09-15T02:58:35.494257+0000","last_became_active":"2024-09-15T02:44:21.628817+0000","last_became_peered":"2024-09-15T02:44:21.628817+0000","last_unstale":"2024-09-15T02:58:35.494257+0000","last_undegraded":"2024-09-15T02:58:35.494257+0000","last_fullsized":"2024-09-15T02:58:35.494257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.494141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.494141+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.494141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:37.857205+0000","scrub_duration":0.0013527649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:36.401887+0000","last_change":"2024-09-15T02:58:36.401860+0000","last_active":"2024-09-15T02:58:36.401887+0000","last_peered":"2024-09-15T02:58:36.401887+0000","last_clean":"2024-09-15T02:58:36.401887+0000","last_became_active":"2024-09-15T02:44:21.583340+0000","last_became_peered":"2024-09-15T02:44:21.583340+0000","last_unstale":"2024-09-15T02:58:36.401887+0000","last_undegraded":"2024-09-15T02:58:36.401887+0000","last_fullsized":"2024-09-15T02:58:36.401887+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:36.401772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:36.401772+0000","last_clean_scrub_stamp":"2024-09-15T02:58:36.401772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:21:35.102014+0000","scrub_duration":0.0017534289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:31.394242+0000","last_change":"2024-09-15T02:58:31.394215+0000","last_active":"2024-09-15T02:58:31.394242+0000","last_peered":"2024-09-15T02:58:31.394242+0000","last_clean":"2024-09-15T02:58:31.394242+0000","last_became_active":"2024-09-15T02:44:21.615991+0000","last_became_peered":"2024-09-15T02:44:21.615991+0000","last_unstale":"2024-09-15T02:58:31.394242+0000","last_undegraded":"2024-09-15T02:58:31.394242+0000","last_fullsized":"2024-09-15T02:58:31.394242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:31.394126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:31.394126+0000","last_clean_scrub_stamp":"2024-09-15T02:58:31.394126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:04.479967+0000","scrub_duration":0.001845719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:34.347239+0000","last_change":"2024-09-15T02:58:34.347204+0000","last_active":"2024-09-15T02:58:34.347239+0000","last_peered":"2024-09-15T02:58:34.347239+0000","last_clean":"2024-09-15T02:58:34.347239+0000","last_became_active":"2024-09-15T02:44:21.588781+0000","last_became_peered":"2024-09-15T02:44:21.588781+0000","last_unstale":"2024-09-15T02:58:34.347239+0000","last_undegraded":"2024-09-15T02:58:34.347239+0000","last_fullsized":"2024-09-15T02:58:34.347239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:34.347107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:34.347107+0000","last_clean_scrub_stamp":"2024-09-15T02:58:34.347107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:50:03.937294+0000","scrub_duration":0.0019575109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:43.410568+0000","last_change":"2024-09-15T02:58:43.410529+0000","last_active":"2024-09-15T02:58:43.410568+0000","last_peered":"2024-09-15T02:58:43.410568+0000","last_clean":"2024-09-15T02:58:43.410568+0000","last_became_active":"2024-09-15T02:44:21.613899+0000","last_became_peered":"2024-09-15T02:44:21.613899+0000","last_unstale":"2024-09-15T02:58:43.410568+0000","last_undegraded":"2024-09-15T02:58:43.410568+0000","last_fullsized":"2024-09-15T02:58:43.410568+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:43.410410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:43.410410+0000","last_clean_scrub_stamp":"2024-09-15T02:58:43.410410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:22.077356+0000","scrub_duration":0.0021423929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1","version":"702'8","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:30.421118+0000","last_change":"2024-09-15T02:58:30.421080+0000","last_active":"2024-09-15T02:58:30.421118+0000","last_peered":"2024-09-15T02:58:30.421118+0000","last_clean":"2024-09-15T02:58:30.421118+0000","last_became_active":"2024-09-15T02:44:21.594670+0000","last_became_peered":"2024-09-15T02:44:21.594670+0000","last_unstale":"2024-09-15T02:58:30.421118+0000","last_undegraded":"2024-09-15T02:58:30.421118+0000","last_fullsized":"2024-09-15T02:58:30.421118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'8","last_scrub_stamp":"2024-09-15T02:58:30.420990+0000","last_deep_scrub":"702'8","last_deep_scrub_stamp":"2024-09-15T02:58:30.420990+0000","last_clean_scrub_stamp":"2024-09-15T02:58:30.420990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:29:09.703086+0000","scrub_duration":0.0037049769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:24.390261+0000","last_change":"2024-09-15T02:58:24.390235+0000","last_active":"2024-09-15T02:58:24.390261+0000","last_peered":"2024-09-15T02:58:24.390261+0000","last_clean":"2024-09-15T02:58:24.390261+0000","last_became_active":"2024-09-15T02:44:21.535920+0000","last_became_peered":"2024-09-15T02:44:21.535920+0000","last_unstale":"2024-09-15T02:58:24.390261+0000","last_undegraded":"2024-09-15T02:58:24.390261+0000","last_fullsized":"2024-09-15T02:58:24.390261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:24.390146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:24.390146+0000","last_clean_scrub_stamp":"2024-09-15T02:58:24.390146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:07:52.094848+0000","scrub_duration":0.001404083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":644,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:19.570618+0000","last_change":"2024-09-15T02:44:21.578890+0000","last_active":"2024-09-15T02:58:19.570618+0000","last_peered":"2024-09-15T02:58:19.570618+0000","last_clean":"2024-09-15T02:58:19.570618+0000","last_became_active":"2024-09-15T02:44:21.578571+0000","last_became_peered":"2024-09-15T02:44:21.578571+0000","last_unstale":"2024-09-15T02:58:19.570618+0000","last_undegraded":"2024-09-15T02:58:19.570618+0000","last_fullsized":"2024-09-15T02:58:19.570618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:44:20.424388+0000","last_clean_scrub_stamp":"2024-09-15T02:44:20.424388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:25.424077+0000","last_change":"2024-09-15T02:58:25.424049+0000","last_active":"2024-09-15T02:58:25.424077+0000","last_peered":"2024-09-15T02:58:25.424077+0000","last_clean":"2024-09-15T02:58:25.424077+0000","last_became_active":"2024-09-15T02:44:21.556078+0000","last_became_peered":"2024-09-15T02:44:21.556078+0000","last_unstale":"2024-09-15T02:58:25.424077+0000","last_undegraded":"2024-09-15T02:58:25.424077+0000","last_fullsized":"2024-09-15T02:58:25.424077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:25.423958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:25.423958+0000","last_clean_scrub_stamp":"2024-09-15T02:58:25.423958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:54:35.916747+0000","scrub_duration":0.0019328279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:31.399204+0000","last_change":"2024-09-15T02:58:31.399177+0000","last_active":"2024-09-15T02:58:31.399204+0000","last_peered":"2024-09-15T02:58:31.399204+0000","last_clean":"2024-09-15T02:58:31.399204+0000","last_became_active":"2024-09-15T02:44:21.582718+0000","last_became_peered":"2024-09-15T02:44:21.582718+0000","last_unstale":"2024-09-15T02:58:31.399204+0000","last_undegraded":"2024-09-15T02:58:31.399204+0000","last_fullsized":"2024-09-15T02:58:31.399204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:31.399083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:31.399083+0000","last_clean_scrub_stamp":"2024-09-15T02:58:31.399083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:49.012848+0000","scrub_duration":0.0013029350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.309743+0000","last_change":"2024-09-15T02:58:35.309690+0000","last_active":"2024-09-15T02:58:35.309743+0000","last_peered":"2024-09-15T02:58:35.309743+0000","last_clean":"2024-09-15T02:58:35.309743+0000","last_became_active":"2024-09-15T02:44:21.584028+0000","last_became_peered":"2024-09-15T02:44:21.584028+0000","last_unstale":"2024-09-15T02:58:35.309743+0000","last_undegraded":"2024-09-15T02:58:35.309743+0000","last_fullsized":"2024-09-15T02:58:35.309743+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.309582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.309582+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.309582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:19:48.590288+0000","scrub_duration":0.001345427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575036,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312084,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059756,"statfs":{"total":96636764160,"available":96317190144,"internally_reserved":0,"allocated":6565888,"data_stored":3647832,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.69399999999999995,"5min":0.75,"15min":1.3959999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60999999999999999,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.28100000000000003},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.80500000000000005},"last":0.68500000000000005}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.55300000000000005},"min":{"1min":0.46100000000000002,"5min":0.33600000000000002,"15min":0.23400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.94599999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59799999999999998,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.27300000000000002},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":1.4430000000000001},"last":0.64700000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60799999999999998,"15min":0.56200000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.27000000000000002},"max":{"1min":0.745,"5min":0.78600000000000003,"15min":0.95899999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.20300000000000001},"max":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":1.478},"last":0.59899999999999998}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57599999999999996,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.28299999999999997},"max":{"1min":0.72699999999999998,"5min":0.83699999999999997,"15min":2.4300000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.57299999999999995},"min":{"1min":0.502,"5min":0.40600000000000003,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.89700000000000002,"15min":1.6120000000000001},"last":0.61899999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45100000000000001,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.24199999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.755},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.28000000000000003},"max":{"1min":0.70599999999999996,"5min":0.83399999999999996,"15min":0.95899999999999996},"last":0.443}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":0.45400000000000001},"min":{"1min":0.35499999999999998,"5min":0.30099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72899999999999998,"15min":0.88900000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":2.2400000000000002},"last":0.47599999999999998}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.77600000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.52500000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.79300000000000004,"5min":0.89700000000000002,"15min":0.92800000000000005},"last":0.45900000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575035,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312036,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059804,"statfs":{"total":96636764160,"available":96317239296,"internally_reserved":0,"allocated":6516736,"data_stored":3118909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.34599999999999997,"15min":0.27000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":5.0250000000000004},"last":0.626},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.874},"min":{"1min":0.46600000000000003,"5min":0.37,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":0.85699999999999998,"15min":71.516999999999996},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59099999999999997,"15min":0.94699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.25900000000000001},"max":{"1min":0.69399999999999995,"5min":0.84199999999999997,"15min":93.381},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56100000000000005,"15min":0.86399999999999999},"min":{"1min":0.39300000000000002,"5min":0.33500000000000002,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.90900000000000003,"15min":79.905000000000001},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.66400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.93899999999999995,"15min":29.305},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.318},"max":{"1min":0.66500000000000004,"5min":0.85499999999999998,"15min":9.8510000000000009},"last":0.56200000000000006}]},{"osd":3,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.69499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.255},"max":{"1min":0.75900000000000001,"5min":0.96599999999999997,"15min":34.877000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.29299999999999998},"max":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":31.657},"last":0.47899999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.51000000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.20300000000000001},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":20.212},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.442,"15min":0.46300000000000002},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.64700000000000002,"5min":0.71599999999999997,"15min":5.6909999999999998},"last":0.36499999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.215},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80000000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.248},"max":{"1min":0.71499999999999997,"5min":0.81799999999999995,"15min":3.012},"last":0.59399999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47099999999999997,"15min":0.59499999999999997},"min":{"1min":0.35099999999999998,"5min":0.32500000000000001,"15min":0.24299999999999999},"max":{"1min":0.68999999999999995,"5min":0.71299999999999997,"15min":28.756},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.48899999999999999,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.32000000000000001,"15min":0.249},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":27.602},"last":0.51700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575037,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307876,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063964,"statfs":{"total":96636764160,"available":96321499136,"internally_reserved":0,"allocated":6516736,"data_stored":3118923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.69899999999999995},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":17.704999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.621,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.78100000000000003,"5min":0.872,"15min":17.465},"last":0.627}]},{"osd":1,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.22900000000000001},"max":{"1min":0.81999999999999995,"5min":0.83999999999999997,"15min":7.1710000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.42999999999999999,"5min":0.33300000000000002,"15min":0.27700000000000002},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":6.7999999999999998},"last":0.501}]},{"osd":2,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1000000000000001,"15min":8.2789999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.60299999999999998},"min":{"1min":0.51600000000000001,"5min":0.33800000000000002,"15min":0.29099999999999998},"max":{"1min":0.79300000000000004,"5min":1.133,"15min":3.0059999999999998},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.628},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.25700000000000001},"max":{"1min":0.81299999999999994,"5min":0.872,"15min":7.8090000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63600000000000001,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.38200000000000001,"15min":0.28599999999999998},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":8.6829999999999998},"last":0.60599999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46300000000000002,"15min":0.44600000000000001},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.216},"max":{"1min":0.68100000000000005,"5min":0.82099999999999995,"15min":0.98699999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.23000000000000001},"max":{"1min":0.68300000000000005,"5min":0.78000000000000003,"15min":2.0750000000000002},"last":0.42699999999999999}]},{"osd":6,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.22800000000000001},"max":{"1min":0.67000000000000004,"5min":0.81799999999999995,"15min":3.4420000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.39100000000000001,"5min":0.33300000000000002,"15min":0.25},"max":{"1min":0.68899999999999995,"5min":0.86299999999999999,"15min":3.0510000000000002},"last":0.44700000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.253},"max":{"1min":0.69699999999999995,"5min":0.79400000000000004,"15min":2.9870000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.94399999999999995,"15min":1.337},"last":0.64800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.86299999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.316},"max":{"1min":0.72099999999999997,"5min":0.873,"15min":0.93100000000000005},"last":0.60799999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.69699999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.32800000000000001},"max":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.379,"15min":0.223},"max":{"1min":0.73699999999999999,"5min":0.89400000000000002,"15min":0.94199999999999995},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.55400000000000005},"min":{"1min":0.47799999999999998,"5min":0.36699999999999999,"15min":0.26800000000000002},"max":{"1min":0.82499999999999996,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60399999999999998,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.36099999999999999},"max":{"1min":0.752,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.55400000000000005}]},{"osd":5,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.23200000000000001},"max":{"1min":0.67400000000000004,"5min":0.69699999999999995,"15min":0.81599999999999995},"last":0.442}]},{"osd":6,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42099999999999999,"15min":0.44700000000000001},"min":{"1min":0.316,"5min":0.31,"15min":0.251},"max":{"1min":0.505,"5min":0.65700000000000003,"15min":0.69699999999999995},"last":0.41499999999999998}]},{"osd":7,"last update":"Sun Sep 15 02:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.78600000000000003,"5min":1.48,"15min":1.48},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.49199999999999999,"15min":0.50600000000000001},"min":{"1min":0.373,"5min":0.31900000000000001,"15min":0.28000000000000003},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.81799999999999995},"last":0.53700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575038,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307876,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063964,"statfs":{"total":96636764160,"available":96321499136,"internally_reserved":0,"allocated":6516736,"data_stored":3118909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36399999999999999,"15min":0.49099999999999999},"min":{"1min":0.25,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.60299999999999998,"5min":0.72399999999999998,"15min":29.562999999999999},"last":0.25},{"interface":"front","average":{"1min":0.435,"5min":0.47599999999999998,"15min":0.44700000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":0.80200000000000005,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44800000000000001,"15min":0.57199999999999995},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.19600000000000001},"max":{"1min":0.69699999999999995,"5min":1.6859999999999999,"15min":35.128},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.432,"5min":0.42599999999999999,"15min":0.57099999999999995},"min":{"1min":0.28499999999999998,"5min":0.26600000000000001,"15min":0.24299999999999999},"max":{"1min":0.71099999999999997,"5min":0.73699999999999999,"15min":29.631},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45600000000000002,"15min":0.60499999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.60599999999999998,"5min":0.90000000000000002,"15min":35.159999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.497,"15min":0.61099999999999999},"min":{"1min":0.377,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.65200000000000002,"5min":0.88600000000000001,"15min":29.603000000000002},"last":0.46000000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.40300000000000002,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":0.80000000000000004,"5min":0.93899999999999995,"15min":7.3499999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.625,"15min":0.78900000000000003},"min":{"1min":0.41699999999999998,"5min":0.40899999999999997,"15min":0.30499999999999999},"max":{"1min":0.78100000000000003,"5min":1.0569999999999999,"15min":35.188000000000002},"last":0.60499999999999998}]},{"osd":5,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.72199999999999998},"min":{"1min":0.40400000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":1.165,"15min":23.082999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.67900000000000005,"15min":0.69499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":1.077,"15min":7.4020000000000001},"last":0.625}]},{"osd":6,"last update":"Sun Sep 15 02:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67300000000000004,"15min":0.78900000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.373},"max":{"1min":0.97699999999999998,"5min":1.365,"15min":35.061999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64600000000000002,"15min":0.74399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44},"max":{"1min":0.90900000000000003,"5min":1.018,"15min":23.129000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65500000000000003,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":1.1060000000000001,"15min":2.1840000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.64800000000000002,"15min":0.80600000000000005},"min":{"1min":0.498,"5min":0.44500000000000001,"15min":0.42099999999999999},"max":{"1min":0.83199999999999996,"5min":0.98199999999999998,"15min":35.100999999999999},"last":0.65100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575036,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312032,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059808,"statfs":{"total":96636764160,"available":96317243392,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.253,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":0.55200000000000005,"5min":0.65000000000000002,"15min":1.2370000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":0.57199999999999995,"5min":0.747,"15min":1.1799999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.27100000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59099999999999997,"5min":0.70999999999999996,"15min":1.319},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.73799999999999999,"15min":1.347},"last":0.53000000000000003}]},{"osd":3,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.34899999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.55900000000000005,"5min":0.66700000000000004,"15min":1.411},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.34599999999999997,"5min":0.29199999999999998,"15min":0.252},"max":{"1min":0.622,"5min":0.67500000000000004,"15min":2.9340000000000002},"last":0.57499999999999996}]},{"osd":4,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.73099999999999998,"5min":0.95499999999999996,"15min":1.298},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.61699999999999999},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.875,"15min":1.1319999999999999},"last":0.81100000000000005}]},{"osd":5,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.315},"max":{"1min":0.81999999999999995,"5min":1,"15min":2.9079999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.49199999999999999,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.94999999999999996,"15min":1.367},"last":0.72199999999999998}]},{"osd":6,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.008,"15min":1.2050000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.96999999999999997,"15min":2.9649999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.50800000000000001,"5min":0.43099999999999999,"15min":0.31900000000000001},"max":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":1.3240000000000001},"last":0.75},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.44600000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":0.82099999999999995,"5min":0.95199999999999996,"15min":1.167},"last":0.77800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575036,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303716,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068124,"statfs":{"total":96636764160,"available":96325758976,"internally_reserved":0,"allocated":6516736,"data_stored":3118923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.38900000000000001,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.17899999999999999},"max":{"1min":0.70399999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51800000000000002,"5min":0.65100000000000002,"15min":0.93100000000000005},"last":0.38400000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56000000000000005,"5min":1.405,"15min":1.405},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.441},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.219},"max":{"1min":0.64700000000000002,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.36599999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.249},"max":{"1min":0.81000000000000005,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.67100000000000004,"5min":2.8490000000000002,"15min":2.8490000000000002},"last":0.40200000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63200000000000001,"15min":0.59999999999999998},"min":{"1min":0.48899999999999999,"5min":0.379,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":1.581,"15min":1.581},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63800000000000001,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.34899999999999998,"15min":0.29299999999999998},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.95499999999999996},"last":0.45800000000000002}]},{"osd":5,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.64200000000000002},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.33300000000000002},"max":{"1min":0.90200000000000002,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.65800000000000003},"min":{"1min":0.56200000000000006,"5min":0.433,"15min":0.371},"max":{"1min":1.004,"5min":1.532,"15min":1.532},"last":0.59099999999999997}]},{"osd":6,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.40400000000000003,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.70299999999999996,"15min":0.66300000000000003},"min":{"1min":0.60899999999999999,"5min":0.46800000000000003,"15min":0.33000000000000002},"max":{"1min":1.0509999999999999,"5min":1.498,"15min":1.498},"last":0.51700000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.53500000000000003,"5min":0.35999999999999999,"15min":0.35699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.32500000000000001},"max":{"1min":0.97399999999999998,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.65600000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307924,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063916,"statfs":{"total":96636764160,"available":96321449984,"internally_reserved":0,"allocated":6565888,"data_stored":3647832,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.503},"min":{"1min":0.251,"5min":0.24199999999999999,"15min":0.182},"max":{"1min":0.64800000000000002,"5min":0.68500000000000005,"15min":14.630000000000001},"last":0.442},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46999999999999997,"15min":0.45100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.222},"max":{"1min":0.69299999999999995,"5min":0.85399999999999998,"15min":2.3570000000000002},"last":0.50900000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44500000000000001,"15min":0.47499999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.20200000000000001},"max":{"1min":0.67000000000000004,"5min":0.93899999999999995,"15min":14.679},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":0.68600000000000005,"15min":1.806},"last":0.42099999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.21099999999999999},"max":{"1min":0.504,"5min":0.73799999999999999,"15min":2.835},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.48899999999999999,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.26300000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":14.558999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64900000000000002,"15min":0.60499999999999998},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.253},"max":{"1min":0.85599999999999998,"5min":0.98599999999999999,"15min":1.5549999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65300000000000002,"15min":0.61699999999999999},"min":{"1min":0.38900000000000001,"5min":0.33200000000000002,"15min":0.30199999999999999},"max":{"1min":0.79000000000000004,"5min":0.94699999999999995,"15min":1.3120000000000001},"last":0.60099999999999998}]},{"osd":5,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63400000000000001,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.32800000000000001},"max":{"1min":0.878,"5min":0.998,"15min":1.1579999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.34300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":7.2969999999999997},"last":0.73499999999999999}]},{"osd":6,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68200000000000005,"15min":0.70599999999999996},"min":{"1min":0.55700000000000005,"5min":0.48599999999999999,"15min":0.35199999999999998},"max":{"1min":0.97499999999999998,"5min":1.0129999999999999,"15min":14.6},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69599999999999995,"15min":0.65400000000000003},"min":{"1min":0.53200000000000003,"5min":0.40999999999999998,"15min":0.35399999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.59499999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67900000000000005,"15min":0.64000000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.36699999999999999},"max":{"1min":0.95399999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.34300000000000003},"max":{"1min":0.97899999999999998,"5min":1.0029999999999999,"15min":1.093},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:58:45.586 INFO:tasks.ceph:pgid 12.1f last_scrub_stamp 2024-09-15T02:44:20.424388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=44, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=58, tm_sec=20, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:58:45.586 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T02:59:05.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:59:05.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.698+0000 7f214459d640 10 monclient: get_monmap_and_config 2024-09-15T02:59:05.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.698+0000 7f214459d640 10 monclient: build_initial_monmap 2024-09-15T02:59:05.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.698+0000 7f214459d640 10 monclient: monmap: 2024-09-15T02:59:05.699 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:59:05.699 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:59:05.699325+0000 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:59:05.699325+0000 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:59:05.700 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.699+0000 7f214459d640 1 Processor -- start 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.699+0000 7f214459d640 1 -- start start 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.699+0000 7f214459d640 10 monclient: init 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.699+0000 7f214459d640 10 monclient: _reopen_session rank -1 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f213c0b2c40 addr v1:172.21.15.104:6789/0 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f213c068c50 0x7f213c068010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient(hunting): picked mon.noname-a con 0x7f213c068c50 addr v2:172.21.15.53:3300/0 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f213c06df40 0x7f213c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient(hunting): picked mon.noname-c con 0x7f213c06df40 addr v2:172.21.15.104:3300/0 2024-09-15T02:59:05.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213c06ec40 con 0x7f213c0b2c40 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213c068550 con 0x7f213c068c50 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213c069cb0 con 0x7f213c06df40 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient(hunting): _renew_subs 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f214459d640 10 monclient(hunting): authenticate will time out at 1682.270097s 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f213c068c50 0x7f213c068010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f213c068c50 0x7f213c068010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55816/0 (socket says 172.21.15.53:55816) 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 -- 172.21.15.53:0/166861062 learned_addr learned my addr 172.21.15.53:0/166861062 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): get_auth_request con 0x7f213c068c50 auth_method 0 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2142b3a640 1 --2- 172.21.15.53:0/166861062 >> v2:172.21.15.104:3300/0 conn(0x7f213c06df40 0x7f213c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2142b3a640 10 monclient(hunting): get_auth_request con 0x7f213c06df40 auth_method 0 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2142b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2142b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2142b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient(hunting): handle_auth_done global_id 26361 payload 274 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 -- 172.21.15.53:0/166861062 >> v2:172.21.15.104:3300/0 conn(0x7f213c06df40 msgr2=0x7f213c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 --2- 172.21.15.53:0/166861062 >> v2:172.21.15.104:3300/0 conn(0x7f213c06df40 0x7f213c06e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 -- 172.21.15.53:0/166861062 >> v1:172.21.15.104:6789/0 conn(0x7f213c0b2c40 legacy=0x7f213c140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:05.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient: _finish_hunting 0 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 monclient: found mon.noname-a 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.700+0000 7f2141b38640 1 -- 172.21.15.53:0/166861062 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c069eb0 con 0x7f213c068c50 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141b38640 1 --2- 172.21.15.53:0/166861062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c068c50 0x7f213c068010 secure :-1 s=READY pgs=5507 cs=0 l=1 rev1=1 crypto rx=0x7f213c0667f0 tx=0x7f212c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0eec9309630ebb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 1 -- 172.21.15.53:0/166861062 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c02ad40 con 0x7f213c068c50 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: dump: 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.707 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: _finish_auth 0 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 1 -- 172.21.15.53:0/166861062 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c032440 con 0x7f213c068c50 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f214459d640 5 monclient: authenticate success, global_id 26361 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f214459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 1 -- 172.21.15.53:0/166861062 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c02aea0 con 0x7f213c068c50 2024-09-15T02:59:05.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f214459d640 10 monclient: get_monmap_and_config success 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f214459d640 4 set_mon_vals no callback set 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.701+0000 7f2141337640 10 monclient: dump: 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:05.709 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 10 monclient: shutdown 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 -- 172.21.15.53:0/166861062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c068c50 msgr2=0x7f213c068010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 --2- 172.21.15.53:0/166861062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c068c50 0x7f213c068010 secure :-1 s=READY pgs=5507 cs=0 l=1 rev1=1 crypto rx=0x7f213c0667f0 tx=0x7f212c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 -- 172.21.15.53:0/166861062 shutdown_connections 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 --2- 172.21.15.53:0/166861062 >> v2:172.21.15.104:3300/0 conn(0x7f213c06df40 0x7f213c06e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 --2- 172.21.15.53:0/166861062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c068c50 0x7f213c068010 unknown :-1 s=CLOSED pgs=5507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 -- 172.21.15.53:0/166861062 >> 172.21.15.53:0/166861062 conn(0x7f213c063810 msgr2=0x7f213c0b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 -- 172.21.15.53:0/166861062 shutdown_connections 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 1 -- 172.21.15.53:0/166861062 wait complete. 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 10 monclient: build_initial_monmap 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.702+0000 7f214459d640 10 monclient: monmap: 2024-09-15T02:59:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 librados: starting msgr at 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 librados: starting objecter 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 Processor -- start 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 -- start start 2024-09-15T02:59:05.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 librados: setting wanted keys 2024-09-15T02:59:05.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 librados: calling monclient init 2024-09-15T02:59:05.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient: init 2024-09-15T02:59:05.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient: _reopen_session rank -1 2024-09-15T02:59:05.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f213c068c50 0x7f213c0a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): picked mon.noname-c con 0x7f213c068c50 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f213c06df40 0x7f213c150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): picked mon.noname-b con 0x7f213c06df40 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c0b2c40 0x7f213c139170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): picked mon.noname-a con 0x7f213c0b2c40 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c0abcf0 con 0x7f213c0b2c40 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c0aa600 con 0x7f213c06df40 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:05.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c0abef0 con 0x7f213c068c50 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): _renew_subs 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.703+0000 7f214459d640 10 monclient(hunting): authenticate will time out at 1682.273968s 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f213c068c50 0x7f213c0a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c0b2c40 0x7f213c139170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f213c068c50 0x7f213c0a98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:37974/0 (socket says 172.21.15.53:37974) 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 1 -- 172.21.15.53:0/3632353120 learned_addr learned my addr 172.21.15.53:0/3632353120 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): get_auth_request con 0x7f213c0b2c40 auth_method 0 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 10 monclient(hunting): get_auth_request con 0x7f213c068c50 auth_method 0 2024-09-15T02:59:05.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f213c06df40 0x7f213c150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 10 monclient(hunting): get_auth_request con 0x7f213c06df40 auth_method 0 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:05.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2141b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient(hunting): handle_auth_done global_id 26364 payload 824 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 -- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f213c068c50 msgr2=0x7f213c0a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f213c068c50 0x7f213c0a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 -- 172.21.15.53:0/3632353120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f213c06df40 msgr2=0x7f213c150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f213c06df40 0x7f213c150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient: _finish_hunting 0 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 monclient: found mon.noname-a 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 -- 172.21.15.53:0/3632353120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c009450 con 0x7f213c0b2c40 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142339640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f213c068c50 0x7f213c0a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2142b3a640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c0b2c40 0x7f213c139170 secure :-1 s=READY pgs=5508 cs=0 l=1 rev1=1 crypto rx=0x7f2134007ed0 tx=0x7f2134009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 client.?.objecter ms_handle_connect 0x7f213c0b2c40 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 1 -- 172.21.15.53:0/3632353120 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134018070 con 0x7f213c0b2c40 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: dump: 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: _finish_auth 0 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 1 -- 172.21.15.53:0/3632353120 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213400ee30 con 0x7f213c0b2c40 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 1 -- 172.21.15.53:0/3632353120 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134014650 con 0x7f213c0b2c40 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f21327fc640 10 monclient: dump: 2024-09-15T02:59:05.721 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 5 monclient: authenticate success, global_id 26364 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 10 monclient: _renew_subs 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213c13dd40 con 0x7f213c0b2c40 2024-09-15T02:59:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f2140b36640 4 set_mon_vals no callback set 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 10 client.26364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 10 monclient: _renew_subs 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213c1514e0 con 0x7f213c0b2c40 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.704+0000 7f214459d640 1 librados: init done 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.706+0000 7f214459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.706+0000 7f214459d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.706+0000 7f214459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.706+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213c0aa3d0 con 0x7f213c0b2c40 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.706+0000 7f21327fc640 1 -- 172.21.15.53:0/3632353120 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f2134003020 con 0x7f213c0b2c40 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f21327fc640 1 --2- 172.21.15.53:0/3632353120 >> v2:172.21.15.104:6816/32804 conn(0x7f212006e800 0x7f2120070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f21327fc640 1 -- 172.21.15.53:0/3632353120 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 502..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f213408bc70 con 0x7f213c0b2c40 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f21327fc640 10 client.26364.objecter ms_dispatch 0x7f213c000b90 osd_map(702..702 src has 502..702) v4 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f21327fc640 3 client.26364.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f21327fc640 3 client.26364.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f2142339640 1 --2- 172.21.15.53:0/3632353120 >> v2:172.21.15.104:6816/32804 conn(0x7f212006e800 0x7f2120070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f2142339640 10 monclient: get_auth_request con 0x7f212006e800 auth_method 0 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f21327fc640 20 client.26364.objecter dump_active .. 0 homeless 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f2142339640 1 --2- 172.21.15.53:0/3632353120 >> v2:172.21.15.104:6816/32804 conn(0x7f212006e800 0x7f2120070cb0 secure :-1 s=READY pgs=3710 cs=0 l=1 rev1=1 crypto rx=0x7f2138001d50 tx=0x7f2138009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.707+0000 7f21327fc640 10 client.26364.objecter ms_handle_connect 0x7f212006e800 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.711+0000 7f21327fc640 1 -- 172.21.15.53:0/3632353120 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f213c0aa3d0 con 0x7f213c0b2c40 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.711+0000 7f21327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.711+0000 7f21327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:59:05.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.872+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 --> v2:172.21.15.104:6816/32804 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f213c0aa3d0 con 0x7f212006e800 2024-09-15T02:59:05.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.876+0000 7f21327fc640 1 -- 172.21.15.53:0/3632353120 <== mgr.4102 v2:172.21.15.104:6816/32804 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+154330 (secure 0 0 0) 0x7f213c0aa3d0 con 0x7f212006e800 2024-09-15T02:59:05.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-15T02:59:05.879 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-15T02:59:05.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.878+0000 7f214459d640 10 librados: watch_flush enter 2024-09-15T02:59:05.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.878+0000 7f214459d640 10 librados: watch_flush exit 2024-09-15T02:59:05.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.878+0000 7f214459d640 20 client.26364.objecter shutdown clearing up homeless session... 2024-09-15T02:59:05.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.878+0000 7f214459d640 10 client.26364.objecter successfully canceled tick 2024-09-15T02:59:05.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.878+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 >> v2:172.21.15.104:6816/32804 conn(0x7f212006e800 msgr2=0x7f2120070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:05.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 --2- 172.21.15.53:0/3632353120 >> v2:172.21.15.104:6816/32804 conn(0x7f212006e800 0x7f2120070cb0 secure :-1 s=READY pgs=3710 cs=0 l=1 rev1=1 crypto rx=0x7f2138001d50 tx=0x7f2138009040 comp rx=0 tx=0).stop 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 10 monclient: shutdown 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c0b2c40 msgr2=0x7f213c139170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c0b2c40 0x7f213c139170 secure :-1 s=READY pgs=5508 cs=0 l=1 rev1=1 crypto rx=0x7f2134007ed0 tx=0x7f2134009850 comp rx=0 tx=0).stop 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 shutdown_connections 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f213c0b2c40 0x7f213c139170 unknown :-1 s=CLOSED pgs=5508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 --2- 172.21.15.53:0/3632353120 >> v2:172.21.15.104:6816/32804 conn(0x7f212006e800 0x7f2120070cb0 unknown :-1 s=CLOSED pgs=3710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f213c06df40 0x7f213c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 --2- 172.21.15.53:0/3632353120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f213c068c50 0x7f213c0a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 >> 172.21.15.53:0/3632353120 conn(0x7f213c063810 msgr2=0x7f213c0ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 shutdown_connections 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 -- 172.21.15.53:0/3632353120 wait complete. 2024-09-15T02:59:05.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.879+0000 7f214459d640 1 librados: shutdown 2024-09-15T02:59:05.891 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":1168,"stamp":"2024-09-15T02:59:05.550986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2471416,"kb_used_data":51000,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752503304,"statfs":{"total":773094113280,"available":770563383296,"internally_reserved":0,"allocated":52224000,"data_stored":26009136,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003216"},"pg_stats":[{"pgid":"12.10","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.309703+0000","last_change":"2024-09-15T02:58:29.309675+0000","last_active":"2024-09-15T02:58:29.309703+0000","last_peered":"2024-09-15T02:58:29.309703+0000","last_clean":"2024-09-15T02:58:29.309703+0000","last_became_active":"2024-09-15T02:44:21.589224+0000","last_became_peered":"2024-09-15T02:44:21.589224+0000","last_unstale":"2024-09-15T02:58:29.309703+0000","last_undegraded":"2024-09-15T02:58:29.309703+0000","last_fullsized":"2024-09-15T02:58:29.309703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.309591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.309591+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.309591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:04:48.245075+0000","scrub_duration":0.001767556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":656,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.249325+0000","last_change":"2024-09-15T02:58:33.249295+0000","last_active":"2024-09-15T02:58:33.249325+0000","last_peered":"2024-09-15T02:58:33.249325+0000","last_clean":"2024-09-15T02:58:33.249325+0000","last_became_active":"2024-09-15T02:44:21.472608+0000","last_became_peered":"2024-09-15T02:44:21.472608+0000","last_unstale":"2024-09-15T02:58:33.249325+0000","last_undegraded":"2024-09-15T02:58:33.249325+0000","last_fullsized":"2024-09-15T02:58:33.249325+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.249211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.249211+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.249211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:54:05.493624+0000","scrub_duration":0.0013138430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:27.193298+0000","last_change":"2024-09-15T02:58:27.193272+0000","last_active":"2024-09-15T02:58:27.193298+0000","last_peered":"2024-09-15T02:58:27.193298+0000","last_clean":"2024-09-15T02:58:27.193298+0000","last_became_active":"2024-09-15T02:44:21.604946+0000","last_became_peered":"2024-09-15T02:44:21.604946+0000","last_unstale":"2024-09-15T02:58:27.193298+0000","last_undegraded":"2024-09-15T02:58:27.193298+0000","last_fullsized":"2024-09-15T02:58:27.193298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:27.193179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:27.193179+0000","last_clean_scrub_stamp":"2024-09-15T02:58:27.193179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:59:07.330194+0000","scrub_duration":0.0015378810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:40.523869+0000","last_change":"2024-09-15T02:58:40.523839+0000","last_active":"2024-09-15T02:58:40.523869+0000","last_peered":"2024-09-15T02:58:40.523869+0000","last_clean":"2024-09-15T02:58:40.523869+0000","last_became_active":"2024-09-15T02:44:21.621686+0000","last_became_peered":"2024-09-15T02:44:21.621686+0000","last_unstale":"2024-09-15T02:58:40.523869+0000","last_undegraded":"2024-09-15T02:58:40.523869+0000","last_fullsized":"2024-09-15T02:58:40.523869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:40.523733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:40.523733+0000","last_clean_scrub_stamp":"2024-09-15T02:58:40.523733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:38:26.536590+0000","scrub_duration":0.001444007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":656,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.285854+0000","last_change":"2024-09-15T02:58:29.285826+0000","last_active":"2024-09-15T02:58:29.285854+0000","last_peered":"2024-09-15T02:58:29.285854+0000","last_clean":"2024-09-15T02:58:29.285854+0000","last_became_active":"2024-09-15T02:44:21.558951+0000","last_became_peered":"2024-09-15T02:44:21.558951+0000","last_unstale":"2024-09-15T02:58:29.285854+0000","last_undegraded":"2024-09-15T02:58:29.285854+0000","last_fullsized":"2024-09-15T02:58:29.285854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.285737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.285737+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.285737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:11:02.269546+0000","scrub_duration":0.0020859030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:37.405564+0000","last_change":"2024-09-15T02:58:37.405540+0000","last_active":"2024-09-15T02:58:37.405564+0000","last_peered":"2024-09-15T02:58:37.405564+0000","last_clean":"2024-09-15T02:58:37.405564+0000","last_became_active":"2024-09-15T02:44:21.592665+0000","last_became_peered":"2024-09-15T02:44:21.592665+0000","last_unstale":"2024-09-15T02:58:37.405564+0000","last_undegraded":"2024-09-15T02:58:37.405564+0000","last_fullsized":"2024-09-15T02:58:37.405564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:37.405455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:37.405455+0000","last_clean_scrub_stamp":"2024-09-15T02:58:37.405455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:23:31.478408+0000","scrub_duration":0.0017888019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:39.369461+0000","last_change":"2024-09-15T02:58:39.369433+0000","last_active":"2024-09-15T02:58:39.369461+0000","last_peered":"2024-09-15T02:58:39.369461+0000","last_clean":"2024-09-15T02:58:39.369461+0000","last_became_active":"2024-09-15T02:44:21.537279+0000","last_became_peered":"2024-09-15T02:44:21.537279+0000","last_unstale":"2024-09-15T02:58:39.369461+0000","last_undegraded":"2024-09-15T02:58:39.369461+0000","last_fullsized":"2024-09-15T02:58:39.369461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:39.369332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:39.369332+0000","last_clean_scrub_stamp":"2024-09-15T02:58:39.369332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:59.952893+0000","scrub_duration":0.0013346740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:44.409095+0000","last_change":"2024-09-15T02:58:44.409072+0000","last_active":"2024-09-15T02:58:44.409095+0000","last_peered":"2024-09-15T02:58:44.409095+0000","last_clean":"2024-09-15T02:58:44.409095+0000","last_became_active":"2024-09-15T02:44:21.612841+0000","last_became_peered":"2024-09-15T02:44:21.612841+0000","last_unstale":"2024-09-15T02:58:44.409095+0000","last_undegraded":"2024-09-15T02:58:44.409095+0000","last_fullsized":"2024-09-15T02:58:44.409095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:44.408977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:44.408977+0000","last_clean_scrub_stamp":"2024-09-15T02:58:44.408977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:29:04.777876+0000","scrub_duration":0.0018179050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:41.561370+0000","last_change":"2024-09-15T02:58:41.561341+0000","last_active":"2024-09-15T02:58:41.561370+0000","last_peered":"2024-09-15T02:58:41.561370+0000","last_clean":"2024-09-15T02:58:41.561370+0000","last_became_active":"2024-09-15T02:44:21.522860+0000","last_became_peered":"2024-09-15T02:44:21.522860+0000","last_unstale":"2024-09-15T02:58:41.561370+0000","last_undegraded":"2024-09-15T02:58:41.561370+0000","last_fullsized":"2024-09-15T02:58:41.561370+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:41.561248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:41.561248+0000","last_clean_scrub_stamp":"2024-09-15T02:58:41.561248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:56:26.494114+0000","scrub_duration":0.0018629569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:24.251933+0000","last_change":"2024-09-15T02:58:24.251914+0000","last_active":"2024-09-15T02:58:24.251933+0000","last_peered":"2024-09-15T02:58:24.251933+0000","last_clean":"2024-09-15T02:58:24.251933+0000","last_became_active":"2024-09-15T02:44:21.535334+0000","last_became_peered":"2024-09-15T02:44:21.535334+0000","last_unstale":"2024-09-15T02:58:24.251933+0000","last_undegraded":"2024-09-15T02:58:24.251933+0000","last_fullsized":"2024-09-15T02:58:24.251933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:24.251846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:24.251846+0000","last_clean_scrub_stamp":"2024-09-15T02:58:24.251846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:16:17.676483+0000","scrub_duration":0.0014797040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:32.440561+0000","last_change":"2024-09-15T02:58:32.440535+0000","last_active":"2024-09-15T02:58:32.440561+0000","last_peered":"2024-09-15T02:58:32.440561+0000","last_clean":"2024-09-15T02:58:32.440561+0000","last_became_active":"2024-09-15T02:44:21.614186+0000","last_became_peered":"2024-09-15T02:44:21.614186+0000","last_unstale":"2024-09-15T02:58:32.440561+0000","last_undegraded":"2024-09-15T02:58:32.440561+0000","last_fullsized":"2024-09-15T02:58:32.440561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:32.440444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:32.440444+0000","last_clean_scrub_stamp":"2024-09-15T02:58:32.440444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:23:30.652467+0000","scrub_duration":0.001723069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.487552+0000","last_change":"2024-09-15T02:58:33.487527+0000","last_active":"2024-09-15T02:58:33.487552+0000","last_peered":"2024-09-15T02:58:33.487552+0000","last_clean":"2024-09-15T02:58:33.487552+0000","last_became_active":"2024-09-15T02:44:21.527331+0000","last_became_peered":"2024-09-15T02:44:21.527331+0000","last_unstale":"2024-09-15T02:58:33.487552+0000","last_undegraded":"2024-09-15T02:58:33.487552+0000","last_fullsized":"2024-09-15T02:58:33.487552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.487436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.487436+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.487436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:55:46.304879+0000","scrub_duration":0.0015127599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":665,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:26.399656+0000","last_change":"2024-09-15T02:58:26.399610+0000","last_active":"2024-09-15T02:58:26.399656+0000","last_peered":"2024-09-15T02:58:26.399656+0000","last_clean":"2024-09-15T02:58:26.399656+0000","last_became_active":"2024-09-15T02:43:21.415861+0000","last_became_peered":"2024-09-15T02:43:21.415861+0000","last_unstale":"2024-09-15T02:58:26.399656+0000","last_undegraded":"2024-09-15T02:58:26.399656+0000","last_fullsized":"2024-09-15T02:58:26.399656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:26.399523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:26.399523+0000","last_clean_scrub_stamp":"2024-09-15T02:58:26.399523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:56:49.179797+0000","scrub_duration":0.001359159,"objects_trimmed":0,"snaptrim_duration":0.000279189,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":663,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:32.753013+0000","last_change":"2024-09-15T02:58:32.752984+0000","last_active":"2024-09-15T02:58:32.753013+0000","last_peered":"2024-09-15T02:58:32.753013+0000","last_clean":"2024-09-15T02:58:32.753013+0000","last_became_active":"2024-09-15T02:44:21.530274+0000","last_became_peered":"2024-09-15T02:44:21.530274+0000","last_unstale":"2024-09-15T02:58:32.753013+0000","last_undegraded":"2024-09-15T02:58:32.753013+0000","last_fullsized":"2024-09-15T02:58:32.753013+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:32.752892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:32.752892+0000","last_clean_scrub_stamp":"2024-09-15T02:58:32.752892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:31:00.482455+0000","scrub_duration":0.0017938310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:28.453457+0000","last_change":"2024-09-15T02:58:28.453430+0000","last_active":"2024-09-15T02:58:28.453457+0000","last_peered":"2024-09-15T02:58:28.453457+0000","last_clean":"2024-09-15T02:58:28.453457+0000","last_became_active":"2024-09-15T02:43:21.413162+0000","last_became_peered":"2024-09-15T02:43:21.413162+0000","last_unstale":"2024-09-15T02:58:28.453457+0000","last_undegraded":"2024-09-15T02:58:28.453457+0000","last_fullsized":"2024-09-15T02:58:28.453457+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:28.453331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:28.453331+0000","last_clean_scrub_stamp":"2024-09-15T02:58:28.453331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:46:28.458820+0000","scrub_duration":0.0017649370000000001,"objects_trimmed":0,"snaptrim_duration":0.000432249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.260014+0000","last_change":"2024-09-15T02:58:35.259984+0000","last_active":"2024-09-15T02:58:35.260014+0000","last_peered":"2024-09-15T02:58:35.260014+0000","last_clean":"2024-09-15T02:58:35.260014+0000","last_became_active":"2024-09-15T02:44:21.579154+0000","last_became_peered":"2024-09-15T02:44:21.579154+0000","last_unstale":"2024-09-15T02:58:35.260014+0000","last_undegraded":"2024-09-15T02:58:35.260014+0000","last_fullsized":"2024-09-15T02:58:35.260014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.259895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.259895+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.259895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:40:55.314158+0000","scrub_duration":0.0015412959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":662,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.440059+0000","last_change":"2024-09-15T02:58:33.440032+0000","last_active":"2024-09-15T02:58:33.440059+0000","last_peered":"2024-09-15T02:58:33.440059+0000","last_clean":"2024-09-15T02:58:33.440059+0000","last_became_active":"2024-09-15T02:43:21.506327+0000","last_became_peered":"2024-09-15T02:43:21.506327+0000","last_unstale":"2024-09-15T02:58:33.440059+0000","last_undegraded":"2024-09-15T02:58:33.440059+0000","last_fullsized":"2024-09-15T02:58:33.440059+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.439945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.439945+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.439945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:32:10.258043+0000","scrub_duration":0.002092979,"objects_trimmed":0,"snaptrim_duration":0.00041126900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.150970+0000","last_change":"2024-09-15T02:58:29.150934+0000","last_active":"2024-09-15T02:58:29.150970+0000","last_peered":"2024-09-15T02:58:29.150970+0000","last_clean":"2024-09-15T02:58:29.150970+0000","last_became_active":"2024-09-15T02:44:21.533501+0000","last_became_peered":"2024-09-15T02:44:21.533501+0000","last_unstale":"2024-09-15T02:58:29.150970+0000","last_undegraded":"2024-09-15T02:58:29.150970+0000","last_fullsized":"2024-09-15T02:58:29.150970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.150843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.150843+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.150843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:01:50.177046+0000","scrub_duration":0.0014478469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:30.510900+0000","last_change":"2024-09-15T02:58:30.510856+0000","last_active":"2024-09-15T02:58:30.510900+0000","last_peered":"2024-09-15T02:58:30.510900+0000","last_clean":"2024-09-15T02:58:30.510900+0000","last_became_active":"2024-09-15T02:43:21.506463+0000","last_became_peered":"2024-09-15T02:43:21.506463+0000","last_unstale":"2024-09-15T02:58:30.510900+0000","last_undegraded":"2024-09-15T02:58:30.510900+0000","last_fullsized":"2024-09-15T02:58:30.510900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:30.510759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:30.510759+0000","last_clean_scrub_stamp":"2024-09-15T02:58:30.510759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:02:35.096214+0000","scrub_duration":0.001387597,"objects_trimmed":0,"snaptrim_duration":0.00051776199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":663,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.720687+0000","last_change":"2024-09-15T02:58:35.720631+0000","last_active":"2024-09-15T02:58:35.720687+0000","last_peered":"2024-09-15T02:58:35.720687+0000","last_clean":"2024-09-15T02:58:35.720687+0000","last_became_active":"2024-09-15T02:44:21.560425+0000","last_became_peered":"2024-09-15T02:44:21.560425+0000","last_unstale":"2024-09-15T02:58:35.720687+0000","last_undegraded":"2024-09-15T02:58:35.720687+0000","last_fullsized":"2024-09-15T02:58:35.720687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.720534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.720534+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.720534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:21:01.695265+0000","scrub_duration":0.0015548070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"702'3","reported_seq":670,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:24.219147+0000","last_change":"2024-09-15T02:58:24.219122+0000","last_active":"2024-09-15T02:58:24.219147+0000","last_peered":"2024-09-15T02:58:24.219147+0000","last_clean":"2024-09-15T02:58:24.219147+0000","last_became_active":"2024-09-15T02:43:21.409788+0000","last_became_peered":"2024-09-15T02:43:21.409788+0000","last_unstale":"2024-09-15T02:58:24.219147+0000","last_undegraded":"2024-09-15T02:58:24.219147+0000","last_fullsized":"2024-09-15T02:58:24.219147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'3","last_scrub_stamp":"2024-09-15T02:58:24.219040+0000","last_deep_scrub":"702'3","last_deep_scrub_stamp":"2024-09-15T02:58:24.219040+0000","last_clean_scrub_stamp":"2024-09-15T02:58:24.219040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:27:46.775672+0000","scrub_duration":0.0028255989999999998,"objects_trimmed":0,"snaptrim_duration":0.00090941399999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:33.356868+0000","last_change":"2024-09-15T02:58:33.356818+0000","last_active":"2024-09-15T02:58:33.356868+0000","last_peered":"2024-09-15T02:58:33.356868+0000","last_clean":"2024-09-15T02:58:33.356868+0000","last_became_active":"2024-09-15T02:44:21.500646+0000","last_became_peered":"2024-09-15T02:44:21.500646+0000","last_unstale":"2024-09-15T02:58:33.356868+0000","last_undegraded":"2024-09-15T02:58:33.356868+0000","last_fullsized":"2024-09-15T02:58:33.356868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:33.356708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:33.356708+0000","last_clean_scrub_stamp":"2024-09-15T02:58:33.356708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:52:18.376144+0000","scrub_duration":0.00155893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":674,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:30.745359+0000","last_change":"2024-09-15T02:58:30.745328+0000","last_active":"2024-09-15T02:58:30.745359+0000","last_peered":"2024-09-15T02:58:30.745359+0000","last_clean":"2024-09-15T02:58:30.745359+0000","last_became_active":"2024-09-15T02:43:21.409808+0000","last_became_peered":"2024-09-15T02:43:21.409808+0000","last_unstale":"2024-09-15T02:58:30.745359+0000","last_undegraded":"2024-09-15T02:58:30.745359+0000","last_fullsized":"2024-09-15T02:58:30.745359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:30.745227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:30.745227+0000","last_clean_scrub_stamp":"2024-09-15T02:58:30.745227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:25:44.715970+0000","scrub_duration":0.0014903729999999999,"objects_trimmed":0,"snaptrim_duration":0.00034760299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:34.449874+0000","last_change":"2024-09-15T02:58:34.449844+0000","last_active":"2024-09-15T02:58:34.449874+0000","last_peered":"2024-09-15T02:58:34.449874+0000","last_clean":"2024-09-15T02:58:34.449874+0000","last_became_active":"2024-09-15T02:44:21.504665+0000","last_became_peered":"2024-09-15T02:44:21.504665+0000","last_unstale":"2024-09-15T02:58:34.449874+0000","last_undegraded":"2024-09-15T02:58:34.449874+0000","last_fullsized":"2024-09-15T02:58:34.449874+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:34.449751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:34.449751+0000","last_clean_scrub_stamp":"2024-09-15T02:58:34.449751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:29:27.629576+0000","scrub_duration":0.0015521109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":661,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:29.525274+0000","last_change":"2024-09-15T02:58:29.525246+0000","last_active":"2024-09-15T02:58:29.525274+0000","last_peered":"2024-09-15T02:58:29.525274+0000","last_clean":"2024-09-15T02:58:29.525274+0000","last_became_active":"2024-09-15T02:43:21.415654+0000","last_became_peered":"2024-09-15T02:43:21.415654+0000","last_unstale":"2024-09-15T02:58:29.525274+0000","last_undegraded":"2024-09-15T02:58:29.525274+0000","last_fullsized":"2024-09-15T02:58:29.525274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:29.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:29.525157+0000","last_clean_scrub_stamp":"2024-09-15T02:58:29.525157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:31:17.681291+0000","scrub_duration":0.001556654,"objects_trimmed":0,"snaptrim_duration":0.00035372000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:32.372925+0000","last_change":"2024-09-15T02:58:32.372896+0000","last_active":"2024-09-15T02:58:32.372925+0000","last_peered":"2024-09-15T02:58:32.372925+0000","last_clean":"2024-09-15T02:58:32.372925+0000","last_became_active":"2024-09-15T02:44:21.644939+0000","last_became_peered":"2024-09-15T02:44:21.644939+0000","last_unstale":"2024-09-15T02:58:32.372925+0000","last_undegraded":"2024-09-15T02:58:32.372925+0000","last_fullsized":"2024-09-15T02:58:32.372925+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:32.372804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:32.372804+0000","last_clean_scrub_stamp":"2024-09-15T02:58:32.372804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:46:40.555477+0000","scrub_duration":0.001735297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"702'2","reported_seq":669,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:27.281177+0000","last_change":"2024-09-15T02:58:27.281138+0000","last_active":"2024-09-15T02:58:27.281177+0000","last_peered":"2024-09-15T02:58:27.281177+0000","last_clean":"2024-09-15T02:58:27.281177+0000","last_became_active":"2024-09-15T02:43:21.412810+0000","last_became_peered":"2024-09-15T02:43:21.412810+0000","last_unstale":"2024-09-15T02:58:27.281177+0000","last_undegraded":"2024-09-15T02:58:27.281177+0000","last_fullsized":"2024-09-15T02:58:27.281177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'2","last_scrub_stamp":"2024-09-15T02:58:27.281056+0000","last_deep_scrub":"702'2","last_deep_scrub_stamp":"2024-09-15T02:58:27.281056+0000","last_clean_scrub_stamp":"2024-09-15T02:58:27.281056+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:54:37.158852+0000","scrub_duration":0.0033533730000000002,"objects_trimmed":0,"snaptrim_duration":0.00040483799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"702'74","reported_seq":749,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:25.589886+0000","last_change":"2024-09-15T02:58:25.589861+0000","last_active":"2024-09-15T02:58:25.589886+0000","last_peered":"2024-09-15T02:58:25.589886+0000","last_clean":"2024-09-15T02:58:25.589886+0000","last_became_active":"2024-09-15T02:43:20.387419+0000","last_became_peered":"2024-09-15T02:43:20.387419+0000","last_unstale":"2024-09-15T02:58:25.589886+0000","last_undegraded":"2024-09-15T02:58:25.589886+0000","last_fullsized":"2024-09-15T02:58:25.589886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'74","last_scrub_stamp":"2024-09-15T02:58:25.589784+0000","last_deep_scrub":"702'74","last_deep_scrub_stamp":"2024-09-15T02:58:25.589784+0000","last_clean_scrub_stamp":"2024-09-15T02:58:25.589784+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:33:24.442211+0000","scrub_duration":0.0044050640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":651,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:38.345943+0000","last_change":"2024-09-15T02:58:38.345914+0000","last_active":"2024-09-15T02:58:38.345943+0000","last_peered":"2024-09-15T02:58:38.345943+0000","last_clean":"2024-09-15T02:58:38.345943+0000","last_became_active":"2024-09-15T02:44:21.629239+0000","last_became_peered":"2024-09-15T02:44:21.629239+0000","last_unstale":"2024-09-15T02:58:38.345943+0000","last_undegraded":"2024-09-15T02:58:38.345943+0000","last_fullsized":"2024-09-15T02:58:38.345943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:38.345813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:38.345813+0000","last_clean_scrub_stamp":"2024-09-15T02:58:38.345813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:51:52.034818+0000","scrub_duration":0.0018938099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:36.447577+0000","last_change":"2024-09-15T02:58:36.447552+0000","last_active":"2024-09-15T02:58:36.447577+0000","last_peered":"2024-09-15T02:58:36.447577+0000","last_clean":"2024-09-15T02:58:36.447577+0000","last_became_active":"2024-09-15T02:44:21.571415+0000","last_became_peered":"2024-09-15T02:44:21.571415+0000","last_unstale":"2024-09-15T02:58:36.447577+0000","last_undegraded":"2024-09-15T02:58:36.447577+0000","last_fullsized":"2024-09-15T02:58:36.447577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:36.447460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:36.447460+0000","last_clean_scrub_stamp":"2024-09-15T02:58:36.447460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:34.347332+0000","scrub_duration":0.001684409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.494257+0000","last_change":"2024-09-15T02:58:35.494232+0000","last_active":"2024-09-15T02:58:35.494257+0000","last_peered":"2024-09-15T02:58:35.494257+0000","last_clean":"2024-09-15T02:58:35.494257+0000","last_became_active":"2024-09-15T02:44:21.628817+0000","last_became_peered":"2024-09-15T02:44:21.628817+0000","last_unstale":"2024-09-15T02:58:35.494257+0000","last_undegraded":"2024-09-15T02:58:35.494257+0000","last_fullsized":"2024-09-15T02:58:35.494257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.494141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.494141+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.494141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:37.857205+0000","scrub_duration":0.0013527649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:36.401887+0000","last_change":"2024-09-15T02:58:36.401860+0000","last_active":"2024-09-15T02:58:36.401887+0000","last_peered":"2024-09-15T02:58:36.401887+0000","last_clean":"2024-09-15T02:58:36.401887+0000","last_became_active":"2024-09-15T02:44:21.583340+0000","last_became_peered":"2024-09-15T02:44:21.583340+0000","last_unstale":"2024-09-15T02:58:36.401887+0000","last_undegraded":"2024-09-15T02:58:36.401887+0000","last_fullsized":"2024-09-15T02:58:36.401887+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:36.401772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:36.401772+0000","last_clean_scrub_stamp":"2024-09-15T02:58:36.401772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:21:35.102014+0000","scrub_duration":0.0017534289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:31.394242+0000","last_change":"2024-09-15T02:58:31.394215+0000","last_active":"2024-09-15T02:58:31.394242+0000","last_peered":"2024-09-15T02:58:31.394242+0000","last_clean":"2024-09-15T02:58:31.394242+0000","last_became_active":"2024-09-15T02:44:21.615991+0000","last_became_peered":"2024-09-15T02:44:21.615991+0000","last_unstale":"2024-09-15T02:58:31.394242+0000","last_undegraded":"2024-09-15T02:58:31.394242+0000","last_fullsized":"2024-09-15T02:58:31.394242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:31.394126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:31.394126+0000","last_clean_scrub_stamp":"2024-09-15T02:58:31.394126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:04.479967+0000","scrub_duration":0.001845719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:34.347239+0000","last_change":"2024-09-15T02:58:34.347204+0000","last_active":"2024-09-15T02:58:34.347239+0000","last_peered":"2024-09-15T02:58:34.347239+0000","last_clean":"2024-09-15T02:58:34.347239+0000","last_became_active":"2024-09-15T02:44:21.588781+0000","last_became_peered":"2024-09-15T02:44:21.588781+0000","last_unstale":"2024-09-15T02:58:34.347239+0000","last_undegraded":"2024-09-15T02:58:34.347239+0000","last_fullsized":"2024-09-15T02:58:34.347239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:34.347107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:34.347107+0000","last_clean_scrub_stamp":"2024-09-15T02:58:34.347107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:50:03.937294+0000","scrub_duration":0.0019575109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:43.410568+0000","last_change":"2024-09-15T02:58:43.410529+0000","last_active":"2024-09-15T02:58:43.410568+0000","last_peered":"2024-09-15T02:58:43.410568+0000","last_clean":"2024-09-15T02:58:43.410568+0000","last_became_active":"2024-09-15T02:44:21.613899+0000","last_became_peered":"2024-09-15T02:44:21.613899+0000","last_unstale":"2024-09-15T02:58:43.410568+0000","last_undegraded":"2024-09-15T02:58:43.410568+0000","last_fullsized":"2024-09-15T02:58:43.410568+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:43.410410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:43.410410+0000","last_clean_scrub_stamp":"2024-09-15T02:58:43.410410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:22.077356+0000","scrub_duration":0.0021423929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1","version":"702'8","reported_seq":668,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:30.421118+0000","last_change":"2024-09-15T02:58:30.421080+0000","last_active":"2024-09-15T02:58:30.421118+0000","last_peered":"2024-09-15T02:58:30.421118+0000","last_clean":"2024-09-15T02:58:30.421118+0000","last_became_active":"2024-09-15T02:44:21.594670+0000","last_became_peered":"2024-09-15T02:44:21.594670+0000","last_unstale":"2024-09-15T02:58:30.421118+0000","last_undegraded":"2024-09-15T02:58:30.421118+0000","last_fullsized":"2024-09-15T02:58:30.421118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'8","last_scrub_stamp":"2024-09-15T02:58:30.420990+0000","last_deep_scrub":"702'8","last_deep_scrub_stamp":"2024-09-15T02:58:30.420990+0000","last_clean_scrub_stamp":"2024-09-15T02:58:30.420990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:29:09.703086+0000","scrub_duration":0.0037049769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:24.390261+0000","last_change":"2024-09-15T02:58:24.390235+0000","last_active":"2024-09-15T02:58:24.390261+0000","last_peered":"2024-09-15T02:58:24.390261+0000","last_clean":"2024-09-15T02:58:24.390261+0000","last_became_active":"2024-09-15T02:44:21.535920+0000","last_became_peered":"2024-09-15T02:44:21.535920+0000","last_unstale":"2024-09-15T02:58:24.390261+0000","last_undegraded":"2024-09-15T02:58:24.390261+0000","last_fullsized":"2024-09-15T02:58:24.390261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:24.390146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:24.390146+0000","last_clean_scrub_stamp":"2024-09-15T02:58:24.390146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:07:52.094848+0000","scrub_duration":0.001404083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":654,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:48.470533+0000","last_change":"2024-09-15T02:58:48.470495+0000","last_active":"2024-09-15T02:58:48.470533+0000","last_peered":"2024-09-15T02:58:48.470533+0000","last_clean":"2024-09-15T02:58:48.470533+0000","last_became_active":"2024-09-15T02:44:21.578571+0000","last_became_peered":"2024-09-15T02:44:21.578571+0000","last_unstale":"2024-09-15T02:58:48.470533+0000","last_undegraded":"2024-09-15T02:58:48.470533+0000","last_fullsized":"2024-09-15T02:58:48.470533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:48.470392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:48.470392+0000","last_clean_scrub_stamp":"2024-09-15T02:58:48.470392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:44:12.026352+0000","scrub_duration":0.002332855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":657,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:25.424077+0000","last_change":"2024-09-15T02:58:25.424049+0000","last_active":"2024-09-15T02:58:25.424077+0000","last_peered":"2024-09-15T02:58:25.424077+0000","last_clean":"2024-09-15T02:58:25.424077+0000","last_became_active":"2024-09-15T02:44:21.556078+0000","last_became_peered":"2024-09-15T02:44:21.556078+0000","last_unstale":"2024-09-15T02:58:25.424077+0000","last_undegraded":"2024-09-15T02:58:25.424077+0000","last_fullsized":"2024-09-15T02:58:25.424077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:25.423958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:25.423958+0000","last_clean_scrub_stamp":"2024-09-15T02:58:25.423958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:54:35.916747+0000","scrub_duration":0.0019328279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:31.399204+0000","last_change":"2024-09-15T02:58:31.399177+0000","last_active":"2024-09-15T02:58:31.399204+0000","last_peered":"2024-09-15T02:58:31.399204+0000","last_clean":"2024-09-15T02:58:31.399204+0000","last_became_active":"2024-09-15T02:44:21.582718+0000","last_became_peered":"2024-09-15T02:44:21.582718+0000","last_unstale":"2024-09-15T02:58:31.399204+0000","last_undegraded":"2024-09-15T02:58:31.399204+0000","last_fullsized":"2024-09-15T02:58:31.399204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:31.399083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:31.399083+0000","last_clean_scrub_stamp":"2024-09-15T02:58:31.399083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:49.012848+0000","scrub_duration":0.0013029350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":658,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-09-15T02:58:35.309743+0000","last_change":"2024-09-15T02:58:35.309690+0000","last_active":"2024-09-15T02:58:35.309743+0000","last_peered":"2024-09-15T02:58:35.309743+0000","last_clean":"2024-09-15T02:58:35.309743+0000","last_became_active":"2024-09-15T02:44:21.584028+0000","last_became_peered":"2024-09-15T02:44:21.584028+0000","last_unstale":"2024-09-15T02:58:35.309743+0000","last_undegraded":"2024-09-15T02:58:35.309743+0000","last_fullsized":"2024-09-15T02:58:35.309743+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:58:35.309582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:58:35.309582+0000","last_clean_scrub_stamp":"2024-09-15T02:58:35.309582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:19:48.590288+0000","scrub_duration":0.001345427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11184,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575040,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312084,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059756,"statfs":{"total":96636764160,"available":96317190144,"internally_reserved":0,"allocated":6565888,"data_stored":3647832,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.72699999999999998,"5min":0.75,"15min":1.3959999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.57699999999999996},"min":{"1min":0.53000000000000003,"5min":0.39100000000000001,"15min":0.28100000000000003},"max":{"1min":0.76500000000000001,"5min":0.77400000000000002,"15min":0.80500000000000005},"last":0.70799999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.55600000000000005},"min":{"1min":0.34899999999999998,"5min":0.33600000000000002,"15min":0.23400000000000001},"max":{"1min":0.745,"5min":0.83099999999999996,"15min":0.94599999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.56899999999999995},"min":{"1min":0.45500000000000002,"5min":0.373,"15min":0.27300000000000002},"max":{"1min":0.70099999999999996,"5min":0.80400000000000005,"15min":1.4430000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.27000000000000002},"max":{"1min":0.77300000000000002,"5min":0.77600000000000002,"15min":0.95899999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56399999999999995,"15min":0.54600000000000004},"min":{"1min":0.435,"5min":0.37,"15min":0.20300000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":1.478},"last":0.68100000000000005}]},{"osd":3,"last update":"Sun Sep 15 02:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56000000000000005,"15min":0.55500000000000005},"min":{"1min":0.442,"5min":0.38700000000000001,"15min":0.28299999999999997},"max":{"1min":0.69899999999999995,"5min":0.83699999999999997,"15min":2.4300000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60999999999999999,"15min":0.57799999999999996},"min":{"1min":0.47999999999999998,"5min":0.40600000000000003,"15min":0.34000000000000002},"max":{"1min":0.753,"5min":0.89700000000000002,"15min":1.6120000000000001},"last":0.54900000000000004}]},{"osd":4,"last update":"Sun Sep 15 02:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44400000000000001,"15min":0.435},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.24199999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.755},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.502,"15min":0.49399999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28000000000000003},"max":{"1min":0.626,"5min":0.70999999999999996,"15min":0.95899999999999996},"last":0.32500000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.315,"5min":0.30099999999999999,"15min":0.24099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.88900000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45300000000000001,"15min":0.46300000000000002},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.22600000000000001},"max":{"1min":0.69099999999999995,"5min":0.75600000000000001,"15min":2.2400000000000002},"last":0.37}]},{"osd":6,"last update":"Sun Sep 15 02:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.32500000000000001,"5min":0.28899999999999998,"15min":0.24399999999999999},"max":{"1min":0.625,"5min":0.74299999999999999,"15min":0.77600000000000002},"last":0.436},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.35399999999999998,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.61599999999999999,"5min":0.84899999999999998,"15min":0.92800000000000005},"last":0.51800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575039,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312036,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059804,"statfs":{"total":96636764160,"available":96317239296,"internally_reserved":0,"allocated":6516736,"data_stored":3118909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.34599999999999997,"15min":0.27000000000000002},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":5.0250000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.58799999999999997,"15min":0.877},"min":{"1min":0.42199999999999999,"5min":0.41199999999999998,"15min":0.28100000000000003},"max":{"1min":0.71899999999999997,"5min":0.85699999999999998,"15min":71.516999999999996},"last":0.60799999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.94999999999999996},"min":{"1min":0.437,"5min":0.35199999999999998,"15min":0.25900000000000001},"max":{"1min":0.69999999999999996,"5min":0.84199999999999997,"15min":93.381},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56000000000000005,"15min":0.86799999999999999},"min":{"1min":0.40100000000000002,"5min":0.33500000000000002,"15min":0.313},"max":{"1min":0.752,"5min":0.90900000000000003,"15min":79.905000000000001},"last":0.57699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.66600000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.30199999999999999},"max":{"1min":0.72399999999999998,"5min":0.93899999999999995,"15min":29.305},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.318},"max":{"1min":0.71499999999999997,"5min":0.85499999999999998,"15min":9.8510000000000009},"last":0.70499999999999996}]},{"osd":3,"last update":"Sun Sep 15 02:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.255},"max":{"1min":0.73899999999999999,"5min":0.96599999999999997,"15min":34.877000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.68300000000000005},"min":{"1min":0.39500000000000002,"5min":0.39100000000000001,"15min":0.29299999999999998},"max":{"1min":0.77200000000000002,"5min":0.80600000000000005,"15min":31.657},"last":0.55500000000000005}]},{"osd":4,"last update":"Sun Sep 15 02:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.433,"15min":0.50700000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.20300000000000001},"max":{"1min":0.64700000000000002,"5min":0.76100000000000001,"15min":20.212},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.60999999999999999,"5min":0.71599999999999997,"15min":5.6909999999999998},"last":0.38100000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.46500000000000002,"15min":0.45900000000000002},"min":{"1min":0.33300000000000002,"5min":0.28299999999999997,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":0.76100000000000001,"15min":0.80000000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.36399999999999999,"15min":0.248},"max":{"1min":0.69699999999999995,"5min":0.81799999999999995,"15min":3.012},"last":0.40699999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46999999999999997,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.24299999999999999},"max":{"1min":0.68500000000000005,"5min":0.71299999999999997,"15min":28.756},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48199999999999998,"15min":0.59999999999999998},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.249},"max":{"1min":0.65900000000000003,"5min":0.78900000000000003,"15min":27.602},"last":0.371}]}]},{"osd":5,"up_from":13,"seq":55834575041,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307876,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063964,"statfs":{"total":96636764160,"available":96321499136,"internally_reserved":0,"allocated":6516736,"data_stored":3118923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.62,"15min":0.70399999999999996},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":0.71099999999999997,"5min":0.93300000000000005,"15min":17.704999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.624,"15min":0.70299999999999996},"min":{"1min":0.432,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.80000000000000004,"5min":0.872,"15min":17.465},"last":0.60899999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.40200000000000002,"5min":0.36699999999999999,"15min":0.22900000000000001},"max":{"1min":0.67300000000000004,"5min":0.83999999999999997,"15min":7.1710000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.27700000000000002},"max":{"1min":0.73299999999999998,"5min":0.85799999999999998,"15min":6.7999999999999998},"last":0.54000000000000004}]},{"osd":2,"last update":"Sun Sep 15 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60499999999999998,"15min":0.61199999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.26400000000000001},"max":{"1min":0.75,"5min":1.1000000000000001,"15min":8.2789999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.65100000000000002,"15min":0.60999999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.29099999999999998},"max":{"1min":0.78500000000000003,"5min":1.133,"15min":3.0059999999999998},"last":0.66200000000000003}]},{"osd":3,"last update":"Sun Sep 15 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60599999999999998,"15min":0.63300000000000001},"min":{"1min":0.46200000000000002,"5min":0.378,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":0.872,"15min":7.8090000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.51900000000000002,"5min":0.41999999999999998,"15min":0.28599999999999998},"max":{"1min":0.79900000000000004,"5min":0.93700000000000006,"15min":8.6829999999999998},"last":0.68300000000000005}]},{"osd":4,"last update":"Sun Sep 15 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.443},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.216},"max":{"1min":0.626,"5min":0.82099999999999995,"15min":0.98699999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.30199999999999999,"5min":0.29799999999999999,"15min":0.23000000000000001},"max":{"1min":0.72499999999999998,"5min":0.78000000000000003,"15min":2.0750000000000002},"last":0.48599999999999999}]},{"osd":6,"last update":"Sun Sep 15 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47799999999999998,"15min":0.46700000000000003},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.22800000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":3.4420000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.25},"max":{"1min":0.65200000000000002,"5min":0.86299999999999999,"15min":3.0510000000000002},"last":0.46500000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.33800000000000002,"5min":0.313,"15min":0.253},"max":{"1min":0.72599999999999998,"5min":0.79400000000000004,"15min":2.9870000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.315,"15min":0.26700000000000002},"max":{"1min":0.78200000000000003,"5min":0.94399999999999995,"15min":1.337},"last":0.42899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575040,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.439,"5min":0.35599999999999998,"15min":0.23400000000000001},"max":{"1min":0.71499999999999997,"5min":0.747,"15min":0.86299999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.59099999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.316},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.93100000000000005},"last":0.51500000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:58:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.55400000000000005},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58299999999999996,"15min":0.54900000000000004},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.32800000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.49399999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:58:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.56399999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.223},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.94199999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.439,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 02:58:47 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60599999999999998,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.36699999999999999,"15min":0.26800000000000002},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.36099999999999999},"max":{"1min":0.77500000000000002,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.61299999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:58:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.33400000000000002,"15min":0.23200000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.81599999999999995},"last":0.40300000000000002}]},{"osd":6,"last update":"Sun Sep 15 02:58:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.36799999999999999,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.59499999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.33200000000000002,"5min":0.31,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.65700000000000003,"15min":0.69699999999999995},"last":0.374}]},{"osd":7,"last update":"Sun Sep 15 02:58:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53300000000000003,"15min":0.51000000000000001},"min":{"1min":0.39200000000000002,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.76100000000000001,"5min":1.48,"15min":1.48},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.504,"15min":0.504},"min":{"1min":0.36399999999999999,"5min":0.32200000000000001,"15min":0.28000000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.45000000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307876,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063964,"statfs":{"total":96636764160,"available":96321499136,"internally_reserved":0,"allocated":6516736,"data_stored":3118909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36399999999999999,"15min":0.49099999999999999},"min":{"1min":0.25,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.60299999999999998,"5min":0.72399999999999998,"15min":29.562999999999999},"last":0.316},{"interface":"front","average":{"1min":0.435,"5min":0.47599999999999998,"15min":0.44700000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":0.80200000000000005,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":0.35599999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44800000000000001,"15min":0.57199999999999995},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.19600000000000001},"max":{"1min":0.69699999999999995,"5min":1.6859999999999999,"15min":35.128},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.42599999999999999,"15min":0.57099999999999995},"min":{"1min":0.28499999999999998,"5min":0.26600000000000001,"15min":0.24299999999999999},"max":{"1min":0.71099999999999997,"5min":0.73699999999999999,"15min":29.631},"last":0.33400000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45600000000000002,"15min":0.60499999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.60599999999999998,"5min":0.90000000000000002,"15min":35.159999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.497,"15min":0.61099999999999999},"min":{"1min":0.377,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.65200000000000002,"5min":0.88600000000000001,"15min":29.603000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.40300000000000002,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":0.80000000000000004,"5min":0.93899999999999995,"15min":7.3499999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.625,"15min":0.78900000000000003},"min":{"1min":0.41699999999999998,"5min":0.40899999999999997,"15min":0.30499999999999999},"max":{"1min":0.78100000000000003,"5min":1.0569999999999999,"15min":35.188000000000002},"last":0.56499999999999995}]},{"osd":5,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.72199999999999998},"min":{"1min":0.40400000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":1.165,"15min":23.082999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.67900000000000005,"15min":0.69499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":1.077,"15min":7.4020000000000001},"last":0.63}]},{"osd":6,"last update":"Sun Sep 15 02:58:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66200000000000003,"15min":0.78900000000000003},"min":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.373},"max":{"1min":0.78600000000000003,"5min":1.117,"15min":35.061999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64800000000000002,"15min":0.74199999999999999},"min":{"1min":0.51200000000000001,"5min":0.45900000000000002,"15min":0.44},"max":{"1min":0.79700000000000004,"5min":1.018,"15min":23.129000000000001},"last":0.52600000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65500000000000003,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":1.1060000000000001,"15min":2.1840000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.64800000000000002,"15min":0.80600000000000005},"min":{"1min":0.498,"5min":0.44500000000000001,"15min":0.42099999999999999},"max":{"1min":0.83199999999999996,"5min":0.98199999999999998,"15min":35.100999999999999},"last":0.59999999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312032,"kb_used_data":6360,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059808,"statfs":{"total":96636764160,"available":96317243392,"internally_reserved":0,"allocated":6512640,"data_stored":3118904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.38,"15min":0.378},"min":{"1min":0.249,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":0.505,"5min":0.65000000000000002,"15min":1.2370000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":0.55700000000000005,"5min":0.747,"15min":1.1799999999999999},"last":0.374}]},{"osd":1,"last update":"Sun Sep 15 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58699999999999997,"5min":0.70999999999999996,"15min":1.319},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.46000000000000002},"min":{"1min":0.378,"5min":0.309,"15min":0.24399999999999999},"max":{"1min":0.58499999999999996,"5min":0.73799999999999999,"15min":1.347},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44,"15min":0.44900000000000001},"min":{"1min":0.374,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.51500000000000001,"5min":0.625,"15min":1.411},"last":0.318},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.252},"max":{"1min":0.61299999999999999,"5min":0.67500000000000004,"15min":2.9340000000000002},"last":0.44400000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.40799999999999997,"15min":0.34000000000000002},"max":{"1min":0.70899999999999996,"5min":0.95499999999999996,"15min":1.298},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.47999999999999998,"5min":0.41699999999999998,"15min":0.34300000000000003},"max":{"1min":0.873,"5min":0.873,"15min":1.1319999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Sun Sep 15 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.46700000000000003,"5min":0.432,"15min":0.315},"max":{"1min":0.80700000000000005,"5min":0.93400000000000005,"15min":2.9079999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.68200000000000005,"15min":0.67000000000000004},"min":{"1min":0.54600000000000004,"5min":0.505,"15min":0.35499999999999998},"max":{"1min":0.82199999999999995,"5min":0.92400000000000004,"15min":1.367},"last":0.70799999999999996}]},{"osd":6,"last update":"Sun Sep 15 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.54300000000000004,"5min":0.42599999999999999,"15min":0.33900000000000002},"max":{"1min":0.78700000000000003,"5min":1.008,"15min":1.2050000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63100000000000001,"15min":0.64100000000000001},"min":{"1min":0.436,"5min":0.40899999999999997,"15min":0.32900000000000001},"max":{"1min":0.78000000000000003,"5min":0.96999999999999997,"15min":2.9649999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.51400000000000001,"5min":0.434,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":0.86699999999999999,"15min":1.3240000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66700000000000004,"15min":0.65900000000000003},"min":{"1min":0.49299999999999999,"5min":0.44600000000000001,"15min":0.41399999999999998},"max":{"1min":0.86099999999999999,"5min":0.95199999999999996,"15min":1.167},"last":0.67100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303716,"kb_used_data":6364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068124,"statfs":{"total":96636764160,"available":96325758976,"internally_reserved":0,"allocated":6516736,"data_stored":3118923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.28399999999999997,"5min":0.216,"15min":0.17899999999999999},"max":{"1min":0.63900000000000001,"5min":0.70399999999999996,"15min":1.4510000000000001},"last":0.378},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.41399999999999998},"min":{"1min":0.30299999999999999,"5min":0.26100000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":0.61799999999999999,"15min":0.93100000000000005},"last":0.46300000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":0.60399999999999998,"5min":0.73599999999999999,"15min":1.405},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42999999999999999,"15min":0.44400000000000001},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.219},"max":{"1min":0.57799999999999996,"5min":0.69299999999999995,"15min":2.8719999999999999},"last":0.497}]},{"osd":3,"last update":"Sun Sep 15 02:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.501,"15min":0.50900000000000001},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.249},"max":{"1min":0.60999999999999999,"5min":0.81000000000000005,"15min":2.8149999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45100000000000001,"15min":0.45800000000000002},"min":{"1min":0.312,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":2.8490000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63400000000000001,"15min":0.60499999999999998},"min":{"1min":0.49299999999999999,"5min":0.39900000000000002,"15min":0.27000000000000002},"max":{"1min":0.875,"5min":0.97799999999999998,"15min":1.581},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.45800000000000002,"5min":0.39600000000000002,"15min":0.29299999999999998},"max":{"1min":0.90900000000000003,"5min":0.94599999999999995,"15min":0.95499999999999996},"last":0.61699999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66700000000000004,"15min":0.64400000000000002},"min":{"1min":0.51200000000000001,"5min":0.438,"15min":0.33300000000000002},"max":{"1min":0.93200000000000005,"5min":0.93700000000000006,"15min":1.7390000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68500000000000005,"15min":0.66000000000000003},"min":{"1min":0.52900000000000003,"5min":0.443,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":1.0620000000000001,"15min":1.532},"last":0.66200000000000003}]},{"osd":6,"last update":"Sun Sep 15 02:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65400000000000003,"15min":0.64400000000000002},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.34599999999999997},"max":{"1min":0.93999999999999995,"5min":1.026,"15min":1.6839999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70199999999999996,"15min":0.66800000000000004},"min":{"1min":0.51700000000000002,"5min":0.46800000000000003,"15min":0.33000000000000002},"max":{"1min":0.97599999999999998,"5min":1.0509999999999999,"15min":1.498},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Sep 15 02:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65000000000000002,"15min":0.64100000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.35699999999999998},"max":{"1min":0.91000000000000003,"5min":0.93400000000000005,"15min":0.98699999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.40400000000000003,"15min":0.32500000000000001},"max":{"1min":0.85599999999999998,"5min":0.97999999999999998,"15min":1.7769999999999999},"last":0.56899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575040,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307924,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063916,"statfs":{"total":96636764160,"available":96321449984,"internally_reserved":0,"allocated":6565888,"data_stored":3647832,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40600000000000003,"15min":0.501},"min":{"1min":0.27100000000000002,"5min":0.251,"15min":0.182},"max":{"1min":0.50700000000000001,"5min":0.68500000000000005,"15min":14.630000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46600000000000003,"15min":0.45600000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.222},"max":{"1min":0.60699999999999998,"5min":0.85399999999999998,"15min":2.3570000000000002},"last":0.42799999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.441,"15min":0.47499999999999998},"min":{"1min":0.26200000000000001,"5min":0.25800000000000001,"15min":0.20200000000000001},"max":{"1min":0.53700000000000003,"5min":0.93899999999999995,"15min":14.679},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.435,"15min":0.433},"min":{"1min":0.247,"5min":0.247,"15min":0.189},"max":{"1min":0.65600000000000003,"5min":0.68600000000000005,"15min":1.806},"last":0.32700000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.21099999999999999},"max":{"1min":0.53200000000000003,"5min":0.73799999999999999,"15min":2.835},"last":0.38},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.48699999999999999,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.30399999999999999,"15min":0.26300000000000001},"max":{"1min":0.63,"5min":0.80400000000000005,"15min":14.558999999999999},"last":0.34999999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.623,"15min":0.60799999999999998},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.253},"max":{"1min":0.81999999999999995,"5min":0.89000000000000001,"15min":1.5549999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.63700000000000001,"15min":0.621},"min":{"1min":0.47099999999999997,"5min":0.33200000000000002,"15min":0.30199999999999999},"max":{"1min":0.73799999999999999,"5min":0.94699999999999995,"15min":1.3120000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Sun Sep 15 02:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.46700000000000003,"5min":0.439,"15min":0.32800000000000001},"max":{"1min":0.68999999999999995,"5min":0.998,"15min":1.1579999999999999},"last":0.624},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76500000000000001,"5min":0.93200000000000005,"15min":7.2969999999999997},"last":0.56599999999999995}]},{"osd":6,"last update":"Sun Sep 15 02:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67400000000000004,"15min":0.70899999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.35199999999999998},"max":{"1min":0.80100000000000005,"5min":1.0129999999999999,"15min":14.6},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68200000000000005,"15min":0.65900000000000003},"min":{"1min":0.48299999999999998,"5min":0.40999999999999998,"15min":0.35399999999999998},"max":{"1min":0.82499999999999996,"5min":1.075,"15min":1.075},"last":0.65400000000000003}]},{"osd":7,"last update":"Sun Sep 15 02:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66600000000000004,"15min":0.64300000000000002},"min":{"1min":0.501,"5min":0.42499999999999999,"15min":0.36699999999999999},"max":{"1min":0.78400000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.66100000000000003,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.39600000000000002,"15min":0.34300000000000003},"max":{"1min":0.73299999999999998,"5min":1.0029999999999999,"15min":1.093},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5592,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:59:05.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T02:59:05.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.990+0000 7f26bd35d640 10 monclient: get_monmap_and_config 2024-09-15T02:59:05.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.990+0000 7f26bd35d640 10 monclient: build_initial_monmap 2024-09-15T02:59:05.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.990+0000 7f26bd35d640 10 monclient: monmap: 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:59:05.991067+0000 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:59:05.991067+0000 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.53:6789/0 mon.noname-a-legacy 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.104:6789/0 mon.noname-c-legacy 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-b 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:3: v2:172.21.15.53:3300/0 mon.noname-a 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:4: v2:172.21.15.104:3300/0 mon.noname-c 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:05.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.990+0000 7f26bd35d640 1 Processor -- start 2024-09-15T02:59:05.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.990+0000 7f26bd35d640 1 -- start start 2024-09-15T02:59:05.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.990+0000 7f26bd35d640 10 monclient: init 2024-09-15T02:59:06.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient: _reopen_session rank -1 2024-09-15T02:59:06.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T02:59:06.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f26b80b0e80 0x7f26b80b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:06.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): picked mon.noname-a con 0x7f26b80b0e80 addr v2:172.21.15.53:3300/0 2024-09-15T02:59:06.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:06.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): picked mon.noname-b con 0x7f26b80b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:59:06.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f26b80a5a00 0x7f26b813e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:06.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): picked mon.noname-c con 0x7f26b80a5a00 addr v2:172.21.15.104:3300/0 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b813d9d0 con 0x7f26b80b0e80 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b80a69c0 con 0x7f26b80a5a00 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80aedf0 con 0x7f26b80b1250 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): _renew_subs 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26bd35d640 10 monclient(hunting): authenticate will time out at 1682.561607s 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f26b80b0e80 0x7f26b80b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:06.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f26b80b0e80 0x7f26b80b0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55834/0 (socket says 172.21.15.53:55834) 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 1 -- 172.21.15.53:0/133309411 learned_addr learned my addr 172.21.15.53:0/133309411 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 10 monclient(hunting): get_auth_request con 0x7f26b80b0e80 auth_method 0 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:37982/0 (socket says 172.21.15.53:37982) 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 10 monclient(hunting): get_auth_request con 0x7f26b80b1250 auth_method 0 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:06.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b77fe640 1 --2- 172.21.15.53:0/133309411 >> v2:172.21.15.104:3300/0 conn(0x7f26b80a5a00 0x7f26b813e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b77fe640 10 monclient(hunting): get_auth_request con 0x7f26b80a5a00 auth_method 0 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.991+0000 7f26b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 10 monclient(hunting): handle_auth_done global_id 20423 payload 274 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 1 -- 172.21.15.53:0/133309411 >> v2:172.21.15.104:3300/0 conn(0x7f26b80a5a00 msgr2=0x7f26b813e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 1 --2- 172.21.15.53:0/133309411 >> v2:172.21.15.104:3300/0 conn(0x7f26b80a5a00 0x7f26b813e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 1 -- 172.21.15.53:0/133309411 >> v2:172.21.15.53:3300/0 conn(0x7f26b80b0e80 msgr2=0x7f26b80b0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 1 --2- 172.21.15.53:0/133309411 >> v2:172.21.15.53:3300/0 conn(0x7f26b80b0e80 0x7f26b80b0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 10 monclient: _finish_hunting 0 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 1 monclient: found mon.noname-b 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.53:3301/0 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 1 -- 172.21.15.53:0/133309411 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b80af240 con 0x7f26b80b1250 2024-09-15T02:59:06.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b6ffd640 1 --2- 172.21.15.53:0/133309411 >> v2:172.21.15.53:3300/0 conn(0x7f26b80b0e80 0x7f26b80b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b67fc640 1 --2- 172.21.15.53:0/133309411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813de60 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009850 tx=0x7f26ac0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d3d0f5ff2e330 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b5ffb640 1 -- 172.21.15.53:0/133309411 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac02abe0 con 0x7f26b80b1250 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.992+0000 7f26b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 10 monclient: dump: 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:06.009 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 10 monclient: _finish_auth 0 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 1 -- 172.21.15.53:0/133309411 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac02ad40 con 0x7f26b80b1250 2024-09-15T02:59:06.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 1 -- 172.21.15.53:0/133309411 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac032670 con 0x7f26b80b1250 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 5 monclient: authenticate success, global_id 20423 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 10 monclient: get_monmap_and_config success 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 4 set_mon_vals no callback set 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26b5ffb640 10 monclient: dump: 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.011 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.012 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:06.012 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:06.012 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:06.012 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:06.012 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:06.012 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:06.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 10 monclient: shutdown 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 -- 172.21.15.53:0/133309411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 msgr2=0x7f26b813de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 --2- 172.21.15.53:0/133309411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813de60 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009850 tx=0x7f26ac0090f0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 -- 172.21.15.53:0/133309411 shutdown_connections 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 --2- 172.21.15.53:0/133309411 >> v2:172.21.15.104:3300/0 conn(0x7f26b80a5a00 0x7f26b813e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 --2- 172.21.15.53:0/133309411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813de60 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 --2- 172.21.15.53:0/133309411 >> v2:172.21.15.53:3300/0 conn(0x7f26b80b0e80 0x7f26b80b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 -- 172.21.15.53:0/133309411 >> 172.21.15.53:0/133309411 conn(0x7f26b8062d90 msgr2=0x7f26b8063190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 -- 172.21.15.53:0/133309411 shutdown_connections 2024-09-15T02:59:06.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.993+0000 7f26bd35d640 1 -- 172.21.15.53:0/133309411 wait complete. 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 10 monclient: build_initial_monmap 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 10 monclient: monmap: 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-a 2024-09-15T02:59:06.014 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-b 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.noname-c 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 1 librados: starting msgr at 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 1 librados: starting objecter 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 1 Processor -- start 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 1 -- start start 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 1 librados: setting wanted keys 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 1 librados: calling monclient init 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.994+0000 7f26bd35d640 10 monclient: init 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient: _reopen_session rank -1 2024-09-15T02:59:06.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b80a5a00 0x7f26b80aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): picked mon.noname-a con 0x7f26b80a5a00 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26b80b0e80 0x7f26b80aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): picked mon.noname-b con 0x7f26b80b0e80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): picked mon.noname-c con 0x7f26b80b1250 addr [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80ab490 con 0x7f26b80a5a00 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80a93d0 con 0x7f26b80b0e80 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80aaf10 con 0x7f26b80b1250 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): _renew_subs 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26bd35d640 10 monclient(hunting): authenticate will time out at 1682.565358s 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b80a5a00 0x7f26b80aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b80a5a00 0x7f26b80aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55836/0 (socket says 172.21.15.53:55836) 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 -- 172.21.15.53:0/1443664131 learned_addr learned my addr 172.21.15.53:0/1443664131 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): get_auth_request con 0x7f26b80a5a00 auth_method 0 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:06.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 10 monclient(hunting): get_auth_request con 0x7f26b80b1250 auth_method 0 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:59:06.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b67fc640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26b80b0e80 0x7f26b80aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b67fc640 10 monclient(hunting): get_auth_request con 0x7f26b80b0e80 auth_method 0 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient(hunting): handle_auth_done global_id 26370 payload 824 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 -- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 msgr2=0x7f26b813a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 -- 172.21.15.53:0/1443664131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26b80b0e80 msgr2=0x7f26b80aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26b80b0e80 0x7f26b80aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:59:06.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 monclient: found mon.noname-a 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.53:3300/0 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 -- 172.21.15.53:0/1443664131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a8009ae0 con 0x7f26b80a5a00 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b77fe640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f26b6ffd640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b80a5a00 0x7f26b80aa1d0 secure :-1 s=READY pgs=5509 cs=0 l=1 rev1=1 crypto rx=0x7f26a800c170 tx=0x7f26a800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f269effd640 10 client.?.objecter ms_handle_connect 0x7f26b80a5a00 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.995+0000 7f269effd640 10 client.?.objecter resend_mon_ops 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a80090d0 con 0x7f26b80a5a00 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:59:06.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: dump: 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: _finish_auth 0 2024-09-15T02:59:06.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a800f040 con 0x7f26b80a5a00 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 5 monclient: authenticate success, global_id 26370 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 10 monclient: _renew_subs 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a80146b0 con 0x7f26b80a5a00 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac009390 con 0x7f26b80a5a00 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:59:06.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f269effd640 10 monclient: dump: 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 7e7cb607-2b58-4a36-a082-99425792b70f 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-09-15T02:42:38.407376+0000 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 17 (quincy) 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:06.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26b57fa640 4 set_mon_vals no callback set 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 10 client.26370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 10 monclient: _renew_subs 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b813ad10 con 0x7f26b80a5a00 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.996+0000 7f26bd35d640 1 librados: init done 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.997+0000 7f26bd35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.997+0000 7f26bd35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.997+0000 7f26bd35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.997+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b8063ff0 con 0x7f26b80a5a00 2024-09-15T02:59:06.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.997+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89611+0+0 (secure 0 0 0) 0x7f26a8014a20 con 0x7f26b80a5a00 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f269effd640 1 --2- 172.21.15.53:0/1443664131 >> v2:172.21.15.104:6816/32804 conn(0x7f26a406e800 0x7f26a4070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(702..702 src has 502..702) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f26a808c4b0 con 0x7f26b80a5a00 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f269effd640 10 client.26370.objecter ms_dispatch 0x7f26b8000b90 osd_map(702..702 src has 502..702) v4 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f269effd640 3 client.26370.objecter handle_osd_map got epochs [702,702] > 0 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f269effd640 3 client.26370.objecter handle_osd_map decoding full epoch 702 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f26b67fc640 1 --2- 172.21.15.53:0/1443664131 >> v2:172.21.15.104:6816/32804 conn(0x7f26a406e800 0x7f26a4070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f26b67fc640 10 monclient: get_auth_request con 0x7f26a406e800 auth_method 0 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f269effd640 20 client.26370.objecter dump_active .. 0 homeless 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f269effd640 10 client.26370.objecter ms_handle_connect 0x7f26a406e800 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:05.998+0000 7f26b67fc640 1 --2- 172.21.15.53:0/1443664131 >> v2:172.21.15.104:6816/32804 conn(0x7f26a406e800 0x7f26a4070cb0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7f26ac034000 tx=0x7f26ac032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:59:06.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.002+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f26a80566a0 con 0x7f26b80a5a00 2024-09-15T02:59:06.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.002+0000 7f269effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:59:06.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.002+0000 7f269effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:59:06.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.166+0000 7f26bd35d640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T02:59:06.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.166+0000 7f26bd35d640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T02:59:06.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.166+0000 7f26bd35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.53:3300/0 2024-09-15T02:59:06.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.166+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f26b80ab1f0 con 0x7f26b80a5a00 2024-09-15T02:59:06.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.170+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f26a80566a0 con 0x7f26b80a5a00 2024-09-15T02:59:06.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.170+0000 7f269effd640 10 monclient: handle_config config(1 keys) v1 2024-09-15T02:59:06.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.170+0000 7f26b57fa640 4 set_mon_vals no callback set 2024-09-15T02:59:06.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.170+0000 7f26b57fa640 10 set_mon_vals mon_health_to_clog = false 2024-09-15T02:59:06.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.170+0000 7f269effd640 1 -- 172.21.15.53:0/1443664131 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f26a8059eb0 con 0x7f26b80a5a00 2024-09-15T02:59:06.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.170+0000 7f269effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T02:59:06.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.170+0000 7f269effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 10 librados: watch_flush enter 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 10 librados: watch_flush exit 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 20 client.26370.objecter shutdown clearing up homeless session... 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 10 client.26370.objecter successfully canceled tick 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 >> v2:172.21.15.104:6816/32804 conn(0x7f26a406e800 msgr2=0x7f26a4070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 --2- 172.21.15.53:0/1443664131 >> v2:172.21.15.104:6816/32804 conn(0x7f26a406e800 0x7f26a4070cb0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7f26ac034000 tx=0x7f26ac032000 comp rx=0 tx=0).stop 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 10 monclient: shutdown 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b80a5a00 msgr2=0x7f26b80aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b80a5a00 0x7f26b80aa1d0 secure :-1 s=READY pgs=5509 cs=0 l=1 rev1=1 crypto rx=0x7f26a800c170 tx=0x7f26a800b040 comp rx=0 tx=0).stop 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 shutdown_connections 2024-09-15T02:59:06.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b80b1250 0x7f26b813a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 --2- 172.21.15.53:0/1443664131 >> v2:172.21.15.104:6816/32804 conn(0x7f26a406e800 0x7f26a4070cb0 unknown :-1 s=CLOSED pgs=3711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26b80b0e80 0x7f26b80aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 --2- 172.21.15.53:0/1443664131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b80a5a00 0x7f26b80aa1d0 unknown :-1 s=CLOSED pgs=5509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:59:06.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 >> 172.21.15.53:0/1443664131 conn(0x7f26b8062d90 msgr2=0x7f26b80707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:59:06.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 shutdown_connections 2024-09-15T02:59:06.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.172+0000 7f26bd35d640 1 -- 172.21.15.53:0/1443664131 wait complete. 2024-09-15T02:59:06.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-15T02:59:06.173+0000 7f26bd35d640 1 librados: shutdown 2024-09-15T02:59:06.184 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T02:59:06.184 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T02:59:06.184 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T02:59:06.184 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.235 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T02:59:06.235 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T02:59:06.235 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.284 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T02:59:06.284 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T02:59:06.284 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.323 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T02:59:06.323 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T02:59:06.324 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.398 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T02:59:06.398 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T02:59:06.398 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.438 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T02:59:06.439 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T02:59:06.439 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.521 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T02:59:06.521 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-15T02:59:06.521 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.564 INFO:tasks.ceph.osd.6:Stopped 2024-09-15T02:59:06.565 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-15T02:59:06.565 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.599 INFO:tasks.ceph.osd.7:Stopped 2024-09-15T02:59:06.599 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T02:59:06.599 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-15T02:59:06.599 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.622 INFO:tasks.ceph.mgr.y:Stopped 2024-09-15T02:59:06.622 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T02:59:06.623 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.669 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T02:59:06.669 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T02:59:06.669 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T02:59:06.669 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.684 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T02:59:06.684 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:59:06.685 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.737 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:59:06.738 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:59:06.738 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:59:06.755 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:59:06.755 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T02:59:06.756 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-09-15T02:59:06.808 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:59:06.808 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T02:59:07.012 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:59:07.012 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T02:59:07.103 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:59:07.103 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T02:59:07.160 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:59:07.160 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T02:59:07.243 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:59:07.243 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T02:59:07.404 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:59:07.404 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T02:59:07.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:59:07.490 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-15T02:59:07.553 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:59:07.553 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-15T02:59:07.639 INFO:tasks.ceph:Archiving mon data... 2024-09-15T02:59:07.641 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/data/mon.a.tgz 2024-09-15T02:59:07.641 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-09-15T02:59:07.659 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.MuENoPYtYB 2024-09-15T02:59:07.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MuENoPYtYB 2024-09-15T02:59:07.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.MuENoPYtYB 2024-09-15T02:59:07.936 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.MuENoPYtYB is 553KB 2024-09-15T02:59:08.040 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.MuENoPYtYB 2024-09-15T02:59:08.057 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/data/mon.c.tgz 2024-09-15T02:59:08.057 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-09-15T02:59:08.111 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.Js6DvMhcOQ 2024-09-15T02:59:08.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Js6DvMhcOQ 2024-09-15T02:59:08.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.Js6DvMhcOQ 2024-09-15T02:59:08.373 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.Js6DvMhcOQ is 553KB 2024-09-15T02:59:08.478 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.Js6DvMhcOQ 2024-09-15T02:59:08.494 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/data/mon.b.tgz 2024-09-15T02:59:08.494 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-15T02:59:08.513 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.1YtHPehDGI 2024-09-15T02:59:08.513 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.1YtHPehDGI 2024-09-15T02:59:08.703 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.1YtHPehDGI 2024-09-15T02:59:08.787 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.1YtHPehDGI is 554KB 2024-09-15T02:59:08.886 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.1YtHPehDGI 2024-09-15T02:59:08.900 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T02:59:08.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:59:08.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:59:08.964 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T02:59:08.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:59:08.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:59:08.997 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:59:09.031 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:59:09.035 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T02:59:09.037 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/remote/smithi053/crash 2024-09-15T02:59:09.038 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:59:09.068 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/remote/smithi104/crash 2024-09-15T02:59:09.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:59:09.099 INFO:tasks.ceph:Compressing logs... 2024-09-15T02:59:09.099 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:59:09.112 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:59:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T02:59:09.135 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39322.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36531.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104830.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39322.log: /var/log/ceph/ceph-client.admin.36531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104872.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39182.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104830.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36531.log.gz 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37922.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104830.log.gz 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104620.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104872.log.gz 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37922.log: /var/log/ceph/ceph-client.admin.38012.log 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104620.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39322.log.gz 2024-09-15T02:59:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36753.log 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104620.log.gz 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37884.log 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38429.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.39182.log.gz 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38627.log 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36753.log.gz 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38318.log 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38627.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T02:59:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38165.log 2024-09-15T02:59:09.145 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.1.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.38012.log.gz 2024-09-15T02:59:09.145 INFO:teuthology.orchestra.run.smithi053.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38318.log.gz 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.37922.log.gz 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104704.log 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37551.log 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104704.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38627.log.gz 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104704.log.gz 2024-09-15T02:59:09.146 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105208.log 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39444.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37551.log.gz 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105377.log 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33155.log 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105208.log.gz/var/log/ceph/ceph-client.admin.105377.log: 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105377.log.gz 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38275.log 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.33155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104914.log 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33155.log.gz 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-09-15T02:59:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38605.log 2024-09-15T02:59:09.154 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39236.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104914.log.gz 2024-09-15T02:59:09.155 INFO:teuthology.orchestra.run.smithi053.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-09-15T02:59:09.156 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38748.log 2024-09-15T02:59:09.156 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36587.log 2024-09-15T02:59:09.156 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38748.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.38275.log.gz 2024-09-15T02:59:09.157 INFO:teuthology.orchestra.run.smithi053.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38748.log.gz 2024-09-15T02:59:09.158 INFO:teuthology.orchestra.run.smithi053.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.38605.log.gz 2024-09-15T02:59:09.158 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38291.log 2024-09-15T02:59:09.158 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38812.log 2024-09-15T02:59:09.158 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36587.log: /var/log/ceph/ceph-client.admin.38291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37304.log 2024-09-15T02:59:09.159 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38812.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36587.log.gz 2024-09-15T02:59:09.159 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39234.log 2024-09-15T02:59:09.160 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37304.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37304.log.gz 2024-09-15T02:59:09.160 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38344.log 2024-09-15T02:59:09.162 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39234.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38812.log.gz 2024-09-15T02:59:09.162 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39132.log 2024-09-15T02:59:09.163 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38344.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.39234.log.gz 2024-09-15T02:59:09.164 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38776.log 2024-09-15T02:59:09.165 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39132.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38344.log.gz 2024-09-15T02:59:09.166 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T02:59:09.170 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33036.log 2024-09-15T02:59:09.170 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-15T02:59:09.170 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5/var/log/ceph/ceph-client.admin.33036.log: --verbose -- /var/log/ceph/ceph-client.admin.33116.log 2024-09-15T02:59:09.170 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log/var/log/ceph/ceph-osd.7.log: 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33036.log.gz 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32996.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33116.log.gz 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.32996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32956.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33247.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32996.log.gz 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.32956.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33247.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.6.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33247.log.gz 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32956.log.gz 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T02:59:09.171 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T02:59:09.172 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33161.log 2024-09-15T02:59:09.177 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38776.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38776.log.gz 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37306.log 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39132.log.gz 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37306.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.38291.log.gz 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37306.log.gz 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36795.log 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39319.log: /var/log/ceph/ceph-client.admin.38775.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.37884.log.gz 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38384.log 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37312.log 2024-09-15T02:59:09.178 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36795.log: /var/log/ceph/ceph-client.admin.38384.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36795.log.gz 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105292.log 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37312.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37312.log.gz 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105166.log 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36612.log 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105292.log: /var/log/ceph/ceph-client.admin.105166.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105292.log.gz 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105166.log.gz 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104535.log 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36612.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36612.log.gz 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38528.log 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-09-15T02:59:09.179 INFO:teuthology.orchestra.run.smithi053.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.38384.log.gz 2024-09-15T02:59:09.181 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104535.log: /var/log/ceph/ceph-client.admin.38528.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104535.log.gz 2024-09-15T02:59:09.181 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37938.log 2024-09-15T02:59:09.181 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38063.log 2024-09-15T02:59:09.183 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.b.log: 85.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T02:59:09.183 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33076.log 2024-09-15T02:59:09.183 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33161.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33161.log.gz 2024-09-15T02:59:09.183 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33207.log 2024-09-15T02:59:09.183 INFO:teuthology.orchestra.run.smithi104.stderr: 92.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T02:59:09.183 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33076.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33076.log.gz 2024-09-15T02:59:09.183 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33207.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33207.log.gz 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39318.log: /var/log/ceph/ceph-client.admin.37938.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36974.log 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38063.log: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.38528.log.gz 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32856.log 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.c.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36974.log.gz 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104662.log 2024-09-15T02:59:09.188 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.32856.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.37938.log.gz 2024-09-15T02:59:09.189 INFO:teuthology.orchestra.run.smithi053.stderr: 90.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104399.log 2024-09-15T02:59:09.189 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32856.log.gz 2024-09-15T02:59:09.189 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105417.log 2024-09-15T02:59:09.189 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104399.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104662.log.gz 2024-09-15T02:59:09.189 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36973.log 2024-09-15T02:59:09.189 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104399.log.gz 2024-09-15T02:59:09.189 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2024-09-15T02:59:09.190 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36973.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105417.log.gz 2024-09-15T02:59:09.191 INFO:teuthology.orchestra.run.smithi053.stderr:gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36973.log.gz 2024-09-15T02:59:09.191 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T02:59:09.191 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37912.log 2024-09-15T02:59:09.192 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.a.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2024-09-15T02:59:09.192 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-09-15T02:59:09.206 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105040.log 2024-09-15T02:59:09.207 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36976.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-09-15T02:59:09.213 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104449.log 2024-09-15T02:59:09.225 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105040.log: 94.7% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-09-15T02:59:09.225 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37311.log 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104449.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104449.log.gz 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104578.log 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105040.log.gz 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37311.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37311.log.gz 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104956.log 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104578.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104578.log.gz 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2024-09-15T02:59:09.231 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104956.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104956.log.gz 2024-09-15T02:59:09.232 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37297.log 2024-09-15T02:59:09.236 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38679.log: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2024-09-15T02:59:09.237 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36711.log 2024-09-15T02:59:09.237 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37297.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37297.log.gz 2024-09-15T02:59:09.237 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38712.log 2024-09-15T02:59:09.238 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36711.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36711.log.gz 2024-09-15T02:59:09.239 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37309.log 2024-09-15T02:59:09.240 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38712.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38712.log.gz 2024-09-15T02:59:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T02:59:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37309.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-09-15T02:59:09.242 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36967.log 2024-09-15T02:59:09.258 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104998.log 2024-09-15T02:59:09.261 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36967.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36967.log.gz 2024-09-15T02:59:09.271 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104746.log 2024-09-15T02:59:09.271 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104998.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104998.log.gz 2024-09-15T02:59:09.285 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26979.log 2024-09-15T02:59:09.286 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104746.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104746.log.gz 2024-09-15T02:59:09.286 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-09-15T02:59:09.286 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.tmp-client.admin.26979.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26979.log.gz 2024-09-15T02:59:09.295 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-09-15T02:59:09.299 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39183.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-09-15T02:59:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37509.log 2024-09-15T02:59:09.315 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39235.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-09-15T02:59:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-15T02:59:09.329 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37509.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37509.log.gz 2024-09-15T02:59:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37307.log 2024-09-15T02:59:09.355 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T02:59:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37307.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37307.log.gz 2024-09-15T02:59:09.363 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37313.log 2024-09-15T02:59:09.375 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36977.log 2024-09-15T02:59:09.375 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37313.log: 94.7% -- replaced with /var/log/ceph/ceph-client.admin.37912.log.gz 2024-09-15T02:59:09.375 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37313.log.gz 2024-09-15T02:59:09.375 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39321.log 2024-09-15T02:59:09.376 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36977.log: 94.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36977.log.gz 2024-09-15T02:59:09.377 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:09.377 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105082.log 2024-09-15T02:59:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39321.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39321.log.gz 2024-09-15T02:59:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104493.log 2024-09-15T02:59:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105082.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105082.log.gz 2024-09-15T02:59:09.380 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37962.log 2024-09-15T02:59:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104493.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104493.log.gz 2024-09-15T02:59:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39181.log 2024-09-15T02:59:09.384 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37962.log: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T02:59:09.384 INFO:teuthology.orchestra.run.smithi053.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.37962.log.gz 2024-09-15T02:59:09.384 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105250.log 2024-09-15T02:59:09.387 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39181.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.39181.log.gz 2024-09-15T02:59:09.388 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104788.log 2024-09-15T02:59:09.389 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105250.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105250.log.gz 2024-09-15T02:59:09.389 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38747.log 2024-09-15T02:59:09.390 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.104788.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104788.log.gz 2024-09-15T02:59:09.391 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39446.log 2024-09-15T02:59:09.394 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38747.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38747.log.gz 2024-09-15T02:59:09.395 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37985.log 2024-09-15T02:59:09.399 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39446.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39446.log.gz 2024-09-15T02:59:09.399 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39445.log 2024-09-15T02:59:09.403 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37985.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.37985.log.gz 2024-09-15T02:59:09.403 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39130.log 2024-09-15T02:59:09.406 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39445.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gz 2024-09-15T02:59:09.407 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36970.log 2024-09-15T02:59:09.410 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39130.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39130.log.gz 2024-09-15T02:59:09.411 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38438.log 2024-09-15T02:59:09.412 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36970.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36970.log.gz 2024-09-15T02:59:09.412 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105124.log 2024-09-15T02:59:09.420 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38438.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.38438.log.gz 2024-09-15T02:59:09.420 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105335.log 2024-09-15T02:59:09.422 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105124.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105124.log.gz 2024-09-15T02:59:09.422 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38815.log 2024-09-15T02:59:09.423 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.105335.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105335.log.gz 2024-09-15T02:59:09.423 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36968.log 2024-09-15T02:59:09.426 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38815.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38815.log.gz 2024-09-15T02:59:09.426 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38040.log 2024-09-15T02:59:09.427 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36968.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36968.log.gz 2024-09-15T02:59:09.427 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2024-09-15T02:59:09.439 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38514.log 2024-09-15T02:59:09.439 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38673.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2024-09-15T02:59:09.453 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2024-09-15T02:59:09.462 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33274.log 2024-09-15T02:59:09.470 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39320.log: 91.2% 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T02:59:09.470 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2024-09-15T02:59:09.477 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.33274.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38514.log.gz 2024-09-15T02:59:09.477 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36975.log 2024-09-15T02:59:09.483 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.3.log: -- replaced with /var/log/ceph/ceph-client.admin.33274.log.gz 2024-09-15T02:59:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39233.log 2024-09-15T02:59:09.491 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36975.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36975.log.gz 2024-09-15T02:59:09.502 INFO:teuthology.orchestra.run.smithi053.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.38040.log.gz 2024-09-15T02:59:09.502 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38215.log 2024-09-15T02:59:09.504 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39233.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39233.log.gz 2024-09-15T02:59:09.505 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2024-09-15T02:59:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.38215.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.38215.log.gz 2024-09-15T02:59:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2024-09-15T02:59:09.513 INFO:teuthology.orchestra.run.smithi104.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T02:59:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.37593.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gz 2024-09-15T02:59:09.516 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39131.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2024-09-15T02:59:09.662 INFO:teuthology.orchestra.run.smithi053.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.38165.log.gz 2024-09-15T02:59:10.117 INFO:teuthology.orchestra.run.smithi104.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T02:59:10.540 INFO:teuthology.orchestra.run.smithi053.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T02:59:10.765 INFO:teuthology.orchestra.run.smithi053.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.38063.log.gz 2024-09-15T02:59:11.539 INFO:teuthology.orchestra.run.smithi053.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T02:59:13.126 INFO:teuthology.orchestra.run.smithi104.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T02:59:13.239 INFO:teuthology.orchestra.run.smithi104.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-15T02:59:13.798 INFO:teuthology.orchestra.run.smithi053.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T02:59:13.833 INFO:teuthology.orchestra.run.smithi053.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T02:59:13.922 INFO:teuthology.orchestra.run.smithi104.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T02:59:14.145 INFO:teuthology.orchestra.run.smithi053.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T02:59:14.409 INFO:teuthology.orchestra.run.smithi104.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-15T02:59:14.411 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-15T02:59:14.411 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m5.254s 2024-09-15T02:59:14.411 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m18.707s 2024-09-15T02:59:14.411 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.617s 2024-09-15T02:59:15.063 INFO:teuthology.orchestra.run.smithi053.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T02:59:15.065 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-15T02:59:15.065 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m5.940s 2024-09-15T02:59:15.065 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m25.332s 2024-09-15T02:59:15.065 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.842s 2024-09-15T02:59:15.065 INFO:tasks.ceph:Archiving logs... 2024-09-15T02:59:15.065 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/remote/smithi053/log 2024-09-15T02:59:15.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:59:18.529 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/remote/smithi104/log 2024-09-15T02:59:18.530 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:59:21.524 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T02:59:21.538 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T02:59:21.539 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:59:21.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:59:21.615 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T02:59:21.624 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T02:59:21.624 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:59:21.628 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:59:21.643 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:59:21.645 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:59:21.648 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:59:21.648 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-09-15T02:59:21.648 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv01.front.sepia.ceph.com 3 6 377 29 +273us[ +274us] +/- 58ms 2024-09-15T02:59:21.648 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -224us[ -224us] +/- 59ms 2024-09-15T02:59:21.648 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -29us[ -29us] +/- 67ms 2024-09-15T02:59:21.648 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:59:21.650 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:59:21.650 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-09-15T02:59:21.650 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 +263us[ +257us] +/- 59ms 2024-09-15T02:59:21.650 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -227us[ -227us] +/- 59ms 2024-09-15T02:59:21.650 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -32us[ -32us] +/- 67ms 2024-09-15T02:59:21.650 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:59:21.651 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T02:59:21.673 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T02:59:21.674 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T02:59:21.757 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T02:59:21.829 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T02:59:21.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T02:59:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:59:21.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T02:59:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:59:21.960 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T02:59:21.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T02:59:21.999 INFO:teuthology.task.internal:Duration was 1363.269757 seconds 2024-09-15T02:59:21.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T02:59:22.048 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T02:59:22.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:59:22.052 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:59:22.095 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T02:59:22.098 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T02:59:22.222 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T02:59:22.222 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-09-15T02:59:22.222 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:59:22.245 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T02:59:22.246 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:59:22.286 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T02:59:22.286 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:59:22.289 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:59:22.354 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T02:59:22.354 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:59:22.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:59:22.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T02:59:22.503 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T02:59:22.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:59:22.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:59:22.557 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T02:59:22.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:22.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:22.594 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-09-15T02:59:22.620 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-09-15T02:59:22.658 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:59:22.674 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:59:22.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T02:59:22.722 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T02:59:22.723 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/remote/smithi053 2024-09-15T02:59:22.723 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:59:22.758 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905170/remote/smithi104 2024-09-15T02:59:22.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:59:22.793 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T02:59:22.793 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:59:22.795 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:59:22.842 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T02:59:22.853 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T02:59:22.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T02:59:22.883 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T02:59:22.883 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:59:22.885 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:59:22.898 INFO:teuthology.orchestra.run.smithi053.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:59 /home/ubuntu/cephtest 2024-09-15T02:59:22.900 INFO:teuthology.orchestra.run.smithi104.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:59 /home/ubuntu/cephtest 2024-09-15T02:59:22.901 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T02:59:22.959 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T02:59:23.101 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} duration: 1363.2697570323944 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T02:59:23.102 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:59:23.234 INFO:teuthology.run:pass